Crawl, Render, Index: Where Technical SEO Problems Actually Begin

A team is four months into a content programme and nothing has moved. The articles are good. The briefs were sensible. Somebody suggests the writing needs more depth, so the next quarter is longer articles, and the quarter after that is more of them. Nobody checks whether the pages were ever fetched, and it turns out a rule in the robots file has been excluding the entire section since the redesign.

That story is not rare, and the reason it keeps happening is structural. A search engine does three separate things to your site, in a fixed order: it crawls, then renders, then indexes. Each stage can fail on its own, each failure produces the same visible outcome, and the first two are invisible from every dashboard a marketing team looks at. So the failure gets attributed to the only stage anybody can see, which is the content.

Technical SEO fundamentals are mostly the discipline of asking, in order, which of the three stages broke before writing a single new word. This piece walks the stages, names the symptom that identifies each one, and gives the sequence we would fix them in.

Why every team skips to the content

Content work is visible. You can see a draft, review it, approve it and watch it appear. Crawling and rendering leave no artefact at all. There is no screen in an analytics tool that says "this page was requested by a crawler and refused", and no email arrives when a template starts emitting the wrong canonical. The evidence exists, but it lives in server logs and in a search console most content teams have never been given access to.

The result is a diagnostic bias with a simple shape: teams fix the layer they can observe. It is not laziness. It is the entirely reasonable behaviour of people working with the instruments they were handed. The fix is not more effort, it is a different first question.

Stage one: crawl

Crawling is discovery and fetching. A crawler finds a URL, checks whether it is permitted to request it, and asks your server for it. Nothing else has happened yet: no JavaScript has run, no ranking has been considered, no decision has been made about quality. The rules governing what a crawler may request are set out in the robots exclusion protocol, and the most expensive misunderstanding in this whole field is thinking that file controls indexing. It does not. It controls fetching.

What actually stops a crawl

• A disallow rule left in place. Usually copied from a staging environment during a launch, and nobody looks at that file again for two years.
• No path to the page. A URL reachable only through a search box, a filter or a script is a URL that may as well not exist. Crawlers follow links in the markup.
• Server responses that are not 200. Timeouts, rate limiting, a firewall that treats an unfamiliar agent as an attacker, or a soft error page returned with a success status.
• Attention spent on junk. Endless filter permutations, session parameters and calendar pages consume requests that would otherwise have gone to pages you care about.
• A sitemap that lies. Redirects, dead URLs and pages marked noindex inside the file teach a crawler to trust it less.

The symptom that identifies it

A crawl failure looks like absence. The page is not merely ranking badly; it is not present in search at all, not even when you search for a distinctive sentence from it in quotes. Your server logs show no request for it from a search crawler, and a URL inspection reports that it has never been fetched. That combination is diagnostic. No other stage produces it, because the other two stages require a fetch to have happened first.

Stage two: render

Once the HTML has been fetched, it may need to be rendered before anybody can tell what is on the page. If the content arrives inside the markup, this stage is trivial and instant. If the markup is a shell and the words are assembled by JavaScript, rendering becomes a separate, queued, resource-hungry job that happens later and can fail on its own terms.

What breaks here

A blocked script file, because a disallow rule covers the directory it lives in. A data request that fails or times out for a client with no cookies and no prior session. Content that only appears after an interaction, since nothing clicks a tab or scrolls a carousel on a crawler’s behalf. A route that renders correctly for a signed-in user and returns an empty state for everybody else. Each of these produces a page that looks perfect in your browser and blank to the thing that matters.

You can remove this stage from the argument entirely, and for content pages we think you usually should. If the words are present in the HTML the server returns, there is nothing to render and nothing to queue. That is a decision about how the site is built rather than an SEO tactic, and it is the reason a structured publishing workspace such as Acrosite generates real files and commits them: the page a crawler receives is already the finished page.

The symptom that identifies it

A render failure looks like a page that exists but is empty. It is indexed, it appears for a search on its own URL, and the snippet is thin, generic or drawn from navigation rather than the body. Fetch the page with JavaScript disabled and you will see exactly what the crawler saw. When the rendered version and the raw HTML disagree about what is on the page, you have found your stage, and the gap between them is the size of the problem.

Stage three: index

Indexing is the decision to store the page and consider it for results. It is the only one of the three stages that involves judgement, and it is where a page can be fetched perfectly, rendered perfectly, and still not be kept. Two families of cause dominate: signals you sent by accident, and a judgement that the page adds nothing.

The accidental signals are the easy half. A noindex directive that survived a staging deployment. A canonical tag pointing at a different page, often a template default that names the homepage. A parameterised URL that duplicates a page already stored. Near-identical pages generated for locations or variants, where the only difference is a word in the heading. All of these are the site instructing search not to keep the page, which is a different problem from the page being unwanted.

The symptom that identifies it

An index failure looks like a page that was fetched and then dropped, or a page whose traffic goes to a different URL than the one you published. Coverage reporting is explicit here in a way it never is about the earlier stages: it will tell you the page was discovered but not indexed, or that it was excluded by a directive, or that another URL was chosen as canonical instead. When the tooling names a reason, believe the reason, and resist the urge to rewrite the page before checking which of the two families you are in.

Reading the symptom back to the stage

In practice you meet the symptom first and have to work backwards. This is the table we keep in our heads when somebody says a page is not performing.

Only the last row is a content problem. Every row above it is a finding an audit should have surfaced, and audits vary enormously in whether they distinguish between these stages or simply list everything they found. Our parent company has written a good companion piece on separating substance from volume in how to read an audit you were sold.

The order to fix things in

The stages are sequential, so the fixes are too. Work upstream first, because a repair at stage three does nothing if stage one is still failing, and because fixing stage one sometimes resolves the other two without further work.

1. Read the robots file, out loud, line by line. This takes four minutes and settles the most consequential question on the list. Do not skim it because you assume it is fine.
2. Confirm the important pages are reachable by link. From the homepage, in markup, in a small number of steps. If a page can only be found through a filter or a search box, it is not discoverable.
3. Check what the server actually returns. Status codes, redirect chains and soft errors returned as successes. A 200 response carrying an error message is worse than an honest 404.
4. Compare raw HTML with the rendered page. One page per template is enough. The differences tell you exactly how much of your content depends on a script running.
5. Audit the directives. Every noindex, every canonical, every alternate. Look for template defaults that were never customised, which is where most accidental exclusions live.
6. Only then look at the writing. By this point you know the page can be fetched, read and kept, so anything still wrong is genuinely about the page itself.

That list is deliberately ordered by dependency rather than by severity, and the two often disagree. A blocked directory scored as a medium finding outranks a critical-sounding schema warning, because one of them prevents everything downstream and the other does not. We have argued that distinction at length in severity or priority, and what to fix first.

A page that was never fetched cannot be improved by better writing. Fix the stage that failed, not the stage you can see.

When it really is the content

Here is the concession, and it matters. Technical work has a ceiling, and plenty of sites are already sitting on it. If your pages are crawled promptly, rendered without JavaScript, indexed with the canonical you intended, and still invisible for the queries you want, then the technical stages have done their job and the answer is genuinely editorial. At that point the useful questions become whether the site covers a subject completely enough to be treated as a source, which is the argument in topical authority and clusters search can read, and whether there is simply enough of it, which we looked at in how many pages a site needs.

The three stages also degrade differently over time. Crawl and index problems tend to arrive in events: a launch, a migration, a plugin, a firewall rule. Render problems arrive gradually as a front end accumulates interactivity. Content problems arrive because a competitor published something better. Knowing which kind of decay you are looking at usually starts with asking what changed on the day the numbers changed, which is the discipline behind every migration checklist worth having.

Where we would start

Get access to the two instruments before you get access to the budget. A search console for the property, and server logs you can filter by user agent. Without those, every conversation about technical SEO is inference, and inference is how a team spends two quarters writing for pages nobody was allowed to fetch.

Then run the six steps above on your ten most commercially important pages rather than on the whole site. Ten is enough to establish whether you have a systemic problem or an isolated one, and it fits in an afternoon. If the ten come back clean, that is a real result: it converts the question from technical to editorial and lets you spend the next quarter on the right thing. If they do not, you now have a ranked list of causes rather than a crawler report with four hundred rows.

One condition reverses this advice. On a brand new site with a handful of pages and no history, the three stages are almost never the problem, and time spent on log analysis is time not spent on having something worth indexing. Build the pages, give them fields that will survive a redesign as we set out in structured content that survives, and start checking the stages once there is enough of a site for something to go quietly wrong in.

More of our writing on the same subject sits in SEO insights. If you have a section that stopped performing and cannot say why, send us the URLs and what changed around that time.