Handling Missing Pages Correctly: Checking 404 Errors and Other Server Responses

Management Summary
A nonexistent website address should lead to an understandable missing-page result rather than a misleading success. Trying random addresses explores how a site handles those situations, including redirects to another page. This matters for visitors navigating broken links and for automated services interpreting the response. The article's sixty-response sample illustrates why status, final content and navigation need to be considered together. A redirect is not automatically wrong, and response size alone cannot establish correct handling.
Contents
A missing page needs a clear message and an appropriate HTTP status. Random test paths reveal how a website handles such requests. Routing rules determine which file or application handles a requested path. Different responses can reflect different routing rules; the difference alone does not establish a contradiction or a configuration error.
On 6 September 2026, twenty domains were tested with three path shapes each. The sixty saved responses form a historical sample, not a report of current behaviour.

Three path shapes and their limits
A random name was requested without an extension, with .html, and with a trailing slash. These forms may reach different file, directory or application rules. Randomness reduces the chance of hitting existing content. It does not guarantee nonexistence or an uncached response: an application can generate content for arbitrary paths.
What the sixty responses show
The raw data contains 37 responses with 404, three with 410, and fourteen redirects: seven 301s, six 302s and one 308. Three responses returned 403 and another three returned 429. Access restrictions and rate limits describe the conditions of this measurement; neither the precise cause nor the normal missing-page response follows from those codes alone.
Sixteen domains returned the same status for all three paths. Eleven consistently returned 404 or 410, three consistently redirected, and two consistently returned 403 or 429. Four domains returned different status codes across the path shapes.
The recorded destinations distinguish the three domains that redirected every probe: Wikipedia redirected to its English host, Mozilla to a language path, and PHP to a search. The raw data does not support the earlier claim of two search destinations. The missing-page probes did not fetch the redirect targets.
No direct 200 does not rule out soft 404s
None of the sixty direct responses had status 200. This does not exclude soft-404 problems behind redirects. A soft 404 can arise when a successful response contains a missing-page message. Google evaluates content; status 200 does not guarantee indexing.
The tool’s vocabulary comparison only indicates similar content. Shared navigation can increase overlap, while a dedicated error page can have little overlap with the front page. The inspected tool flags every direct 200 as a suspected soft 404, regardless of the similarity score. Confirmation requires checking the content and any redirect destination.
Distinguishing 404 from 410
404 means that no current representation was found, or that the server is unwilling to disclose existence. It does not specify whether the condition is permanent. 410 describes a resource that is no longer available and is likely to remain unavailable. RFC 9110 recommends 404 when permanence is unknown.
Google treats 404 and 410 alike for the processing described here. A general promise that major crawlers stop sooner after 410 is therefore unsupported. In the sample, this website returned three 410 responses with 408 bytes of content each. Short is not empty. The tool’s threshold of fewer than 512 bytes proves neither absence of content nor absence of navigation.
Separating size from indexing
Sixteen of the forty 404/410 responses exceeded 100 KiB in the recorded evaluation. Four were stored with a rounded size of 256 KiB, matching the reading limit of 262,144 bytes. Without complete response data and a retained truncation flag, an exact total size cannot be inferred. The measurement covered the captured response body, not the whole page cost including images, scripts and stylesheets.
Useful error pages can provide navigation or search. Their size should be proportionate to that benefit. A claim that every crawler reads only the status code would be too broad.
The tool detected no noindex markers in these responses. However, its simple HTML check misses cases such as reversed meta-attribute order. Non-detection therefore does not establish absence. A genuinely missing page needs an appropriate error status; noindex does not replace that meaning. A valid page intended to stay out of search results can legitimately use noindex.
Making the tested base address explicit
The checker first follows the reference page’s redirects, then builds a base from the final scheme, host and port. Random paths start at that host’s root: a reference page under /en/ therefore produces a probe such as /probe-123 rather than /en/probe-123. This also applies to other language or application paths. This avoids certain preliminary host redirects but does not automatically test every section of a website.
The earlier comparison of 49 redirects with 11 cannot be reproduced from the available measurement file. That file contains fourteen redirects. Comparable repeat measurements require the same scope and retrieval conditions, with intermediate destinations retained.
A bounded curl test
The following Bash example requires curl and OpenSSL. The example domain is a placeholder. A fresh random name is requested in three forms; the response headers also expose any Location destination. Redirects are initially not followed. Retrieval failures are reported separately and must not be interpreted as HTTP status 000.
base="https://www.example.com"
name="probe-$(openssl rand -hex 12)" || exit 1
for path in "$name" "$name.html" "$name/"; do
printf '\n%s\n' "$path"
if ! curl --silent --show-error --connect-timeout 5 --max-time 15 \
--dump-header - --output /dev/null \
--write-out 'HTTP %{http_code}; body bytes %{size_download}\n' \
"$base/$path"; then
printf 'Retrieval failed\n' >&2
fi
doneThe Soft 404 and Error Page Checker adds size and vocabulary comparisons. The limitations described above remain relevant to its results.
Questions and answers
Is every redirect of a random path an error?
No. A host or language redirect can be normal. The destination, final status and content determine the interpretation. Redirecting everything to an unrelated front page does not reliably resolve a missing resource.
What does a persistent 429 mean?
Google handles 429 similarly to server errors and temporarily reduces crawling. Persistent errors can eventually cause affected URLs to leave the index. A single measurement proves neither that outcome nor a permanent restriction.


