{"id":15753,"date":"2026-11-10T08:05:00","date_gmt":"2026-11-10T07:05:00","guid":{"rendered":"https:\/\/www.lukaswojcik.com\/blog\/?p=15753"},"modified":"2026-10-26T13:16:06","modified_gmt":"2026-10-26T12:16:06","slug":"robots-txt-matching-rules-and-interpreting-a-twenty-domain-sample","status":"publish","type":"post","link":"https:\/\/www.lukaswojcik.com\/blog\/en\/web-development\/tutorials-en-web-development\/robots-txt-matching-rules-and-interpreting-a-twenty-domain-sample\/","title":{"rendered":"robots.txt: Matching Rules and Interpreting a Twenty-Domain Sample"},"content":{"rendered":"

robots.txt gives instructions to cooperating crawlers, programs that automatically fetch web pages. Evaluation requires three separate checks: the result of fetching the file, the applicable crawler groups, and the most specific matching path rule. The first matching line does not automatically win, and repeated matching groups cannot simply be discarded.<\/p>\n

A historical test from 6 September 2026 covered twenty domains, eight identifiers and eight paths. Rechecking the archive and analyzer reveals limits that matter: the saved matrix is an old tool result, not proof of actual crawler behavior or a site-wide block.<\/p>\n

\"Applicable
Applicable groups must be combined; historical HTTP responses and eight test paths require careful interpretation.<\/figcaption><\/figure>\n

Selection of the applicable rules<\/h2>\n

RFC 9309 matches the crawler’s product token without case sensitivity and requires combining multiple matching groups. Google additionally documents selection of the most specific applicable user agent; repeated groups for that agent are merged. A specific group is not combined with the fallback *<\/code> group. A longest-prefix shortcut applied to an arbitrary HTTP User-Agent string is not a complete description of the standard.<\/p>\n

Inside the applicable rules, the most specific matching path wins. Matching starts at the beginning of the path and compares bytes, called octets, rather than simply counting visible characters. Percent-encoded forms such as %62 for the letter b need the normalization required by the protocol before comparison. For equivalent Allow and Disallow rules, Allow is preferred. Paths are case-sensitive in normal matching. *<\/code> matches zero or more characters; $<\/code> anchors the end.<\/p>\n

An empty Disallow:<\/code> is ignored. It does not undo another Disallow in the same applicable rules. The following isolated example allows \/private\/help.html<\/code> but disallows other paths starting with \/private\/<\/code>, including \/private\/report.html<\/code>:<\/p>\n

User-agent: ExampleBot\nDisallow: \/private\/\nDisallow:\n\nUser-agent: ExampleBot\nAllow: \/private\/help.html$\n<\/code><\/pre>\n
The second ExampleBot group contributes the exception. A parser keeping only the first group produces the wrong result. Blank lines or comments between consecutive user-agent declarations must not accidentally detach the later rules.<\/p>\n
What the historical run actually shows<\/h2>\n
The archive records eighteen HTTP 200 responses, one 410 from heise.de and one 418 from stackoverflow.com. Sixteen starting URLs involved redirects. The alternate www.heise.de check returned 200; the www.stackoverflow.com check also returned 418. The recorded Stack Overflow response was 113 bytes, not empty. Neither observation identifies the operator’s intention or proves a redirect configuration defect.<\/p>\n
The old matrix marked all eight test paths disallowed for GPTBot and CCBot on six domains, ClaudeBot on five, and Google-Extended, PerplexityBot and Applebot-Extended on four each. Those counts describe the stored tool output. Eight paths cannot establish that every URL on a site is disallowed, and the parser defects below prevent treating the old classifications as independently verified provider policy.<\/p>\n
Fourteen domains had no such all-eight-path result for any of the six identifiers. Twelve had no disallowed test path for any of them. These are different counts: google.com and microsoft.com had partial restrictions in the stored output. The original chart mixed these categories.<\/p>\n
The summary reports group counts up to 105 for the main checks and a largest file of 28,275 bytes. Counts are parser-dependent. The returned sitemap list was capped at twenty entries, so a stored value of twenty does not establish exactly twenty declarations. A byte order mark also does not identify the editor that produced a file.<\/p>\n
The fetch status comes before rule interpretation<\/h2>\n
A successful response makes parseable rules relevant. RFC 9309 allows crawling when robots.txt is unavailable, for example after a 4xx response; this is permission under the protocol, not proof that a file physically does not exist. Network and server errors instead require an assumption of complete disallow, with caching and prolonged-failure provisions. The standard gives thirty days as an example of a prolonged period, not a universal timer.<\/p>\n
Google documents a specific handling sequence: server errors initially stop crawling for twelve hours; subsequent behavior depends on the last good cached file and overall site availability. Google treats 429 differently from other 4xx responses. A single status code therefore does not determine crawler behavior independently of time, cache and implementation.<\/p>\n
Host, scheme and port define the scope. A robots.txt redirect can lead to another host while its rules still apply to the original authority. A firewall should allow the intended robots.txt response to reach legitimate crawlers, but robots.txt never replaces authentication or other access controls.<\/p>\n
<\/p>\n
\"Ducks
Ducks paddle freely in the open water and avoid the cracked ice completely. Likewise: if rules are missing, search engines may fetch everything; if the server reports an error, everything counts as blocked.<\/figcaption><\/figure>\n
Six identifiers do not mean six training crawlers<\/h2>\n
GPTBot and ClaudeBot concern their providers’ model-development collection. CCBot builds Common Crawl’s open web archive. PerplexityBot supports search discovery, not foundation-model training according to its provider. Separate search or user-triggered agents can have different rules.<\/p>\n
Google-Extended has no separate HTTP user agent. Its control covers specified Gemini training and grounding uses without changing Google Search inclusion or ranking. Applebot-Extended also does not crawl independently; it controls use of Applebot data for foundation-model training. Neither token is a blanket control over every AI feature or every request.<\/p>\n
Limits of the linked matrix<\/h2>\n
The robots.txt Crawler Matrix<\/a> displays up to eight identifiers against twelve paths and shows a winning line. The inspected implementation, however, keeps only one matching group, mishandles some blank-line group boundaries, lacks required percent-normalization and can display allowed cells after a network failure. Its general 4xx treatment also does not reproduce Google’s 429 behavior. These are concrete limits, not differences left unspecified by the RFC.<\/p>\n
The historical summaries do not retain the complete rule files, so a corrected historical evaluation cannot be reconstructed from those summaries alone. Current deployment checks need the actual file, status, redirect chain, relevant product tokens and representative paths, compared with the crawler provider’s documented behavior.<\/p>\n
\n
Questions and answers<\/h2>\n
Does Disallow remove an address from search?<\/h3>\n
No reliable guarantee. Crawling and indexing are separate. Google must be able to fetch a page to see a noindex meta tag or X-Robots-Tag. Private content requires access controls, not merely a crawl directive.<\/p>\n
Does a 200 HTML error page mean there are no rules?<\/h3>\n
Not necessarily. Google attempts to extract valid rules and ignores other lines. Status, content type and body all need inspection; an HTML wrapper is not a reliable substitute for a valid UTF-8 text file.<\/p>\n
Is Crawl-delay a universal setting?<\/h3>\n
No. It is outside the core protocol and support varies. Google does not support it; Anthropic documents support. Limits and controls must be checked for the intended provider.<\/p>\n
Do changes apply immediately?<\/h3>\n
Not necessarily. Cached robots.txt files and provider processing introduce delay. The RFC generally discourages cache use beyond twenty-four hours unless the file is unreachable. Testing a parser does not prove that a crawler has already fetched the new version.<\/p>\n<\/div>\n
\n
Sources<\/h2>\n