LW IT Solutions logo LW IT Solutions logo LW IT Solutions
« Blog Overview /Digital Marketing/Tutorials / Handling Missing Pages Correctly: Checking 404 Errors...
Read this article in other languages:

Handling Missing Pages Correctly: Checking 404 Errors and Other Server Responses

Handling Missing Pages Correctly: Checking 404 Errors and Other Server ResponsesHandling Missing Pages Correctly: Checking 404 Errors and Other Server Responses

Management Summary

A nonexistent website address should lead to an understandable missing-page result rather than a misleading success. Trying random addresses explores how a site handles those situations, including redirects to another page. This matters for visitors navigating broken links and for automated services interpreting the response. The article's sixty-response sample illustrates why status, final content and navigation need to be considered together. A redirect is not automatically wrong, and response size alone cannot establish correct handling.

Contents
  1. Three path shapes and their limits
  2. What the sixty responses show
  3. No direct 200 does not rule out soft 404s
  4. Distinguishing 404 from 410
  5. Separating size from indexing
  6. Making the tested base address explicit
  7. A bounded curl test
  8. Questions and answers
  9. Sources

A missing page needs a clear message and an appropriate HTTP status. Random test paths reveal how a website handles such requests. Routing rules determine which file or application handles a requested path. Different responses can reflect different routing rules; the difference alone does not establish a contradiction or a configuration error.

On 6 September 2026, twenty domains were tested with three path shapes each. The sixty saved responses form a historical sample, not a report of current behaviour.

Sixty historical responses: forty errors, fourteen redirects and six restricted responses
Historical sample from 6 September 2026. Redirect targets were not fetched.

Three path shapes and their limits

A random name was requested without an extension, with .html, and with a trailing slash. These forms may reach different file, directory or application rules. Randomness reduces the chance of hitting existing content. It does not guarantee nonexistence or an uncached response: an application can generate content for arbitrary paths.

What the sixty responses show

The raw data contains 37 responses with 404, three with 410, and fourteen redirects: seven 301s, six 302s and one 308. Three responses returned 403 and another three returned 429. Access restrictions and rate limits describe the conditions of this measurement; neither the precise cause nor the normal missing-page response follows from those codes alone.

Sixteen domains returned the same status for all three paths. Eleven consistently returned 404 or 410, three consistently redirected, and two consistently returned 403 or 429. Four domains returned different status codes across the path shapes.

The recorded destinations distinguish the three domains that redirected every probe: Wikipedia redirected to its English host, Mozilla to a language path, and PHP to a search. The raw data does not support the earlier claim of two search destinations. The missing-page probes did not fetch the redirect targets.

No direct 200 does not rule out soft 404s

None of the sixty direct responses had status 200. This does not exclude soft-404 problems behind redirects. A soft 404 can arise when a successful response contains a missing-page message. Google evaluates content; status 200 does not guarantee indexing.

The tool’s vocabulary comparison only indicates similar content. Shared navigation can increase overlap, while a dedicated error page can have little overlap with the front page. The inspected tool flags every direct 200 as a suspected soft 404, regardless of the similarity score. Confirmation requires checking the content and any redirect destination.

Distinguishing 404 from 410

404 means that no current representation was found, or that the server is unwilling to disclose existence. It does not specify whether the condition is permanent. 410 describes a resource that is no longer available and is likely to remain unavailable. RFC 9110 recommends 404 when permanence is unknown.

Google treats 404 and 410 alike for the processing described here. A general promise that major crawlers stop sooner after 410 is therefore unsupported. In the sample, this website returned three 410 responses with 408 bytes of content each. Short is not empty. The tool’s threshold of fewer than 512 bytes proves neither absence of content nor absence of navigation.

Separating size from indexing

Sixteen of the forty 404/410 responses exceeded 100 KiB in the recorded evaluation. Four were stored with a rounded size of 256 KiB, matching the reading limit of 262,144 bytes. Without complete response data and a retained truncation flag, an exact total size cannot be inferred. The measurement covered the captured response body, not the whole page cost including images, scripts and stylesheets.

Useful error pages can provide navigation or search. Their size should be proportionate to that benefit. A claim that every crawler reads only the status code would be too broad.

The tool detected no noindex markers in these responses. However, its simple HTML check misses cases such as reversed meta-attribute order. Non-detection therefore does not establish absence. A genuinely missing page needs an appropriate error status; noindex does not replace that meaning. A valid page intended to stay out of search results can legitimately use noindex.

Making the tested base address explicit

The checker first follows the reference page’s redirects, then builds a base from the final scheme, host and port. Random paths start at that host’s root: a reference page under /en/ therefore produces a probe such as /probe-123 rather than /en/probe-123. This also applies to other language or application paths. This avoids certain preliminary host redirects but does not automatically test every section of a website.

The earlier comparison of 49 redirects with 11 cannot be reproduced from the available measurement file. That file contains fourteen redirects. Comparable repeat measurements require the same scope and retrieval conditions, with intermediate destinations retained.

A bounded curl test

The following Bash example requires curl and OpenSSL. The example domain is a placeholder. A fresh random name is requested in three forms; the response headers also expose any Location destination. Redirects are initially not followed. Retrieval failures are reported separately and must not be interpreted as HTTP status 000.

Shell
base="https://www.example.com"
name="probe-$(openssl rand -hex 12)" || exit 1
for path in "$name" "$name.html" "$name/"; do
  printf '\n%s\n' "$path"
  if ! curl --silent --show-error --connect-timeout 5 --max-time 15 \
      --dump-header - --output /dev/null \
      --write-out 'HTTP %{http_code}; body bytes %{size_download}\n' \
      "$base/$path"; then
    printf 'Retrieval failed\n' >&2
  fi
done

The Soft 404 and Error Page Checker adds size and vocabulary comparisons. The limitations described above remain relevant to its results.

Questions and answers

Is every redirect of a random path an error?

No. A host or language redirect can be normal. The destination, final status and content determine the interpretation. Redirecting everything to an unrelated front page does not reliably resolve a missing resource.

What does a persistent 429 mean?

Google handles 429 similarly to server errors and temporarily reduces crawling. Persistent errors can eventually cause affected URLs to leave the index. A single measurement proves neither that outcome nor a permanent restriction.

Lukas Wojcik

Lukas Wojcik

Systems architect and technology enthusiast specializing in scalable tracking solutions, GMP Stack (GA4 & GTM), and robust backend architectures. Advocate for clean code and privacy-first design.

Get in touch

A question or an idea for a web project? Tell me a little about it. This form uses reCAPTCHA and formconfirmed to protect against spam.

Write a comment

Practical experience and questions about the implementation are welcome here.

The email address is not published. Required fields are marked with an asterisk.

LUKASWOJCIK.COM · NEWSLETTER

Fresh ideas in your inbox

New articles, practical tools and updates from the projects.

Choose topics (optional)
No selection: all topics.

Subscription starts after confirmation by email. · Privacy

Privacy · Newsletter

Email address, language, selected topics and subscription/confirmation dates are stored to deliver the newsletter. Double opt-in is required. Every issue includes an immediate unsubscribe link. Formconfirmed checks the signup for spam. No email open or click tracking.

Read the full privacy policy ↗