HTTP Status Codes: Tell Crawlers What Happened

Read the response before judging the page

A browser's visual output does not tell you the response status. A beautifully designed not-found page may arrive with 200, while a helpful maintenance notice may correctly arrive with a temporary server error. Use developer tools or an HTTP client to inspect headers and status for the actual URL. When diagnosing a redirect, preserve the intermediate responses as well as the final page. Otherwise a tool that follows redirects automatically can conceal the path a request took.

HTTP Semantics, RFC 9110, defines the meaning of response codes. Their role is communication about a resource and request, not an editorial score. A 200 response says the request succeeded under its semantics; it does not certify accuracy, originality or usefulness. Google's crawler guidance likewise distinguishes successful retrieval from indexing. Keep that boundary clear when investigating a page that loads normally but fails to appear in search.

Use the same request method that represents the problem you are investigating. A quick header check can be convenient, but some applications handle HEAD differently from GET. If a tool reports an unexpected status, inspect an ordinary GET response before concluding that readers receive the same result. Also note where the response originates: a hosting edge, reverse proxy and application can each generate errors. Knowing that boundary prevents the team from changing application routing when the request never reached the application in the first place.

Use success only when the resource is present

An ordinary published article should return its actual content with a successful response. Beware of application shells that return success for every path regardless of whether an article exists. They may later show a missing message after a browser request fails. To a system inspecting the initial response, the site has reported a resource successfully. The resulting ambiguity can complicate diagnosis, especially when the same title and thin shell appear across thousands of invalid addresses.

An illustrative test is to request a valid article and a nonsense slug within the same section. Compare their status, document title and content. If both return 200 and the same shell, investigate routing and data resolution. A missing resource should not inherit a real article's canonical address simply to make the template complete. Frameworks have different handling, including streaming constraints, so check the deployed response. Next.js SEO explains a specific case where the visible not-found component alone is insufficient evidence.

Redirect genuine moves to the closest replacement

Use a permanent redirect when the resource has permanently moved and there is an appropriate destination. For public reading pages, both 301 and 308 communicate permanence, while their request method behaviour differs. Choose according to your application's HTTP requirements rather than treating the numbers as interchangeable in every context. A temporary redirect fits a temporary move. Avoid issuing a permanent move for a short experiment that you expect to reverse, because clients and intermediaries may remember it.

Map an old article to the page that now fulfils its purpose. If a guide about importing a file moves to a new slug, the replacement is straightforward. If several overlapping guides are consolidated, ensure the destination retains the useful answer that made each old page relevant. Redirecting every removed URL to the home page does not preserve meaning. The editorial decision about what survives belongs in content consolidation; the response code should express that decision accurately.

Keep redirect paths short and coherent

A chain can arise gradually: an HTTP address moves to HTTPS, an old host moves to a new host and an old slug moves to a new slug. Each hop adds another request and another configuration point that can fail. Where feasible, send old public addresses directly to their final intended destination. Also update internal links, canonical metadata and sitemap entries to use that destination. Continuing to advertise the old route makes the redirect part of normal navigation instead of a compatibility measure.

Test for loops and accidental broad rules. A pattern intended to redirect one article family may also catch the destination, producing an endless cycle. Query parameters and trailing slashes can create subtle variants. Use a small explicit mapping for editorial moves when that is easier to inspect than a clever wildcard. During a content migration, preserve a list of old addresses and expected outcomes so the team can confirm the live behaviour after deployment rather than relying on configuration review alone.

Distinguish missing content from an outage

A 404 describes a resource that is not found. A 410 states that access to the resource is no longer available and is likely permanent. Use the distinction when it reflects what you know, without promising that one produces a particular search outcome or removal speed. A missing article can still offer useful navigation, a search field and a clear explanation. Those helpful elements do not require a success status. The page should help the person recover while accurately reporting the missing resource.

An outage is different. If a database fails and an existing article cannot be assembled, returning 404 wrongly describes a temporary infrastructure problem as missing content. A suitable server error communicates failure, and a 503 can describe temporary unavailability. Where applicable, Retry-After can indicate a retry time. Do not leave a maintenance response in place indefinitely. Monitor recovery and confirm normal content returns after the service is restored, including through caches or edge servers that may still hold an earlier response.

Handle access restrictions intentionally

Authentication and permission responses belong to resources that actually require access control. A public learning article accidentally returning an authentication screen is an access defect, even if the developer can view it while signed in. Rate limits and security challenges can similarly affect automated requests differently from a familiar browser session. Investigate the reason for the response before changing content. Crawlability provides the wider access investigation, including cases where a firewall or application rule prevents normal public retrieval.

Do not make a sitemap your only monitor for these failures. A listed URL can return an access restriction, a redirect or an error. Sample public routes from outside the authenticated editorial environment and log meaningful status changes over time. Separate expected missing pages from widespread server failures in reporting. A few obsolete requests do not have the same operational significance as every current article returning an error. Grouping everything into a single count labelled SEO errors hides that distinction.

Turn status checks into clear release decisions

For each route class, define the expected result: published article, moved article, removed article and temporary failure. Test representative cases during technical release checks, including direct navigation to nested routes. Inspect both status and body, since a correct code with an empty or misleading page still deserves attention. Keep canonical URLs and discovery files consistent with those outcomes. This modest response contract makes incidents easier to diagnose and prevents editorial teams from rewriting content to solve a server communication problem.

Sources and further reading

• HTTP Semantics, RFC 9110Defines HTTP response semantics including successful, redirect and error responses.
• Google crawler HTTP status guidanceSuccessful responses do not guarantee indexing; status classes have different crawling consequences.