Why a website redesign should start with the old URL map

Six weeks after a relaunch, organic traffic is down by a third and nobody can say why. The new site is better in every visible respect. It is faster, it looks current, the content is stronger.

The cause is almost never the design. It is that the new site did not inherit what the old one had earned, and that outcome was decided before a single page was built.

What a site accumulates that a redesign can discard

An established site has assets that are invisible in a design review:

• URLs that rank. Frequently for queries nobody on the current team targeted or remembers.
• Inbound links. A directory listing, a supplier page, a news mention. Each points at a specific URL, and each stops passing anything if that URL 404s.
• Crawl history. Search engines have a model of your site’s structure. Changing all of it at once forces that model to be rebuilt.
• Pages that convert quietly. An old service page from 2019 that produces two enquiries a month and that nobody has looked at since.

None of these appear in the brief. All of them are at risk.

The inventory comes before the design conversation

Not after. Before, because it changes the plan. Every time we have run this, at least one page turns out to be earning steady traffic that nobody had mentioned.

Crawl the existing site

# Everything the sitemap declares
curl -s https://example.com/sitemap.xml | grep -oE '<loc>[^<]+' | sed 's/<loc>//' > urls.txt

# Status, title and canonical for each
while read -r u; do
  code=$(curl -sL -o /tmp/p.html -w '%{http_code}' "$u")
  title=$(grep -oiE '<title>[^<]*' /tmp/p.html | head -1 | sed 's/<title>//')
  echo "$u|$code|$title"
done < urls.txt > inventory.csv

A sitemap is a claim about what should exist, not a record of what does. Also crawl your internal links, and if the site is WordPress, check for URLs the sitemap plugin excludes.

Get the data that actually matters

The crawl tells you what exists. Search Console tells you what is worth keeping.

Export the last 16 months of Search Console pages data, and analytics landing page data over the same period. Then join them to the crawl.

That join is the deliverable. It turns “we have 340 pages” into “these 40 pages produce 80 percent of the organic entries, these 60 have inbound links, and these 180 have had no organic entrance in a year.”

Decide per page, with evidence

Four outcomes:

Decision	When	Action at launch
Keep the URL	The page continues to exist and the URL is reasonable	No redirect needed
Move	Content continues at a different address	301 to the new URL
Merge	Several thin pages become one stronger page	301 all of them to the merged page
Retire	No traffic, no links, no purpose	410, or 301 only if a genuine intent-equivalent exists

The single most useful principle: do not change a URL without a reason better than tidiness. Changing a URL has a real cost and needs to buy something.

The mistakes that cause the drop

Redirecting everything to the homepage

This is the worst one, and it is common because it is easy.

Google treats a redirect to an irrelevant page as a soft 404. The target inherits nothing. You have converted a page that ranked into a page that does not exist, with extra steps.

Every URL needs an intent-equivalent destination. If there genuinely is not one, 410 is a more honest answer than pretending the homepage is a substitute.

Deleting content because it looked dated

A content cull that runs on impression rather than data. The 2019 service page nobody likes is removed. It was producing enquiries.

Check performance before deleting. Always.

Changing the URL structure for aesthetics

/services/vps-migration/ becoming /what-we-do/infrastructure/vps-migration/ gains you nothing and costs a redirect on every page plus a period of reassessment.

Restructure when the existing structure genuinely cannot express what the site now needs. Not because a new structure is more logical to whoever is building it.

Redirect chains

/old-page to /interim-page to /new-page. Each hop costs a round trip for users and dilutes the signal. If you have redirected before, resolve chains to point directly at the final destination.

# Count the hops
curl -sIL -o /dev/null -w '%{num_redirects} hops -> %{url_effective}\n' https://example.com/old-page

Staging that got indexed

The staging site is crawled, duplicates the live site, and occasionally outranks it. Then when the real site launches, the duplicate has to be untangled.

Block staging with HTTP authentication rather than robots.txt alone. robots.txt prevents crawling, not indexing, and a URL discovered through a link can be indexed without ever being crawled.

Nobody watches after launch

The project is declared complete at deployment. Problems that would have been trivial in week one are found in month three.

The redirect map is a real artifact

Not a spreadsheet assembled the night before launch. A file with a reason column, produced during the project, tested before it.

old_url,old_status,new_url,redirect_code,reason
/services/hosting/,200,/services/managed-vps-support/,301,Intent equivalent
/blog/2019/wordpress-tips/,200,/insights/wordpress-performance/,301,Merged into stronger page
/team/,200,,410,Content retired, no equivalent
/old-landing-page/,200,/services/web-development/,301,Ranked for web development queries

The reason column is what makes it reviewable. Six months later, when someone asks why an old URL goes where it does, the answer is in the file.

Test it against staging before launch

# Every old URL, against the new site, before DNS changes
while IFS=, read -r old status new code reason; do
  [ "$old" = "old_url" ] && continue
  result=$(curl -sI -o /dev/null -w '%{http_code} %{redirect_url}' "https://staging.example.com$old")
  echo "$old -> $result (expected $code $new)"
done < redirect-map.csv

Every line should produce the expected status and destination. This catches the pattern-matching error where a rule intended for one path catches a hundred.

Launch day and the weeks after

At launch:

• Redirects deployed with the site, not after it
• New sitemap submitted
• Staging protection confirmed still in place
• A crawl of the live site to confirm status codes match expectations

Then watch, for at least a month:

• Search Console index coverage, specifically new exclusions
• Crawl errors, which surface missed redirects within days
• Server logs for 404s with a referrer, which is the fastest way to find a URL you missed
• Organic entrances by landing page, compared against the same period before launch

# 404s that something actually linked to, which are the ones that matter
awk '$9 == 404 && $11 != "\"-\"" {print $7, $11}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn | head -30

That command is worth running weekly for the first month. It finds the URLs your inventory missed, because a real user or crawler just tried one.

What to expect even when it goes well

Some movement is normal. Search engines have to recrawl and reassess, and rankings can fluctuate for a few weeks. Google’s own guidance on site moves describes this as expected.

What is not normal is a large drop that does not recover. That has a cause, it is almost always one of the items above, and it is almost always visible in Search Console within the first fortnight if somebody is looking.

The short version

A redesign is two projects that must be done together: changing what the site says, and migrating what it has already earned. Teams plan the first thoroughly and discover the second at launch.

Start with the inventory. It costs a few days, it frequently changes the plan, and it is the difference between a relaunch that compounds what you had and one that starts over.