English
Deutsch
Polskirobots.txt Fetch Frequency: What Historical Logs Can and Cannot Prove
Management Summary
Search crawlers revisit robots.txt to obtain a website's access instructions. Server records can show those visits, which helps explore how often automated services look for updated rules. A recorded fetch does not establish when the rules were applied, and historical totals without precise time periods cannot promise an update deadline. This matters when access needs to change quickly. Average fetch frequency is not a protective barrier; urgent restrictions need controls that actually enforce access or request limits.
Contents
A robots.txt change is a published instruction, not an access-control switch. A request in an access log shows contact with the server; it does not by itself prove that a crawler received, understood and applied the new rules. The retained analysis is useful, but its method limits the conclusions.
The retained evidence
The original article describes 22 August to 4 September 2026. The saved output contains fourteen log-file groups, and the script selects the current Apache log plus thirteen rotated files. Rotation moves older log entries into separate files; a file does not necessarily cover one day. The script does not filter timestamps. Fourteen files therefore do not independently establish fourteen complete days or the stated date range.
The revised historical script includes both apex and www host prefixes. It groups requests by text in the User-Agent, without verifying source IPs. These are claimed identities, not authenticated crawler operators. It also counts any request line containing robots.txt, including a nested path, a backup filename or a query value. Status and method are not checked.
What the saved counts actually say
The table below reproduces the saved aggregation. The last column divides the count by fourteen to make the original arithmetic comparable; it is not a newly verified daily rate. Requests other than those matching robots.txt are not necessarily pages: redirects, errors and assets may be included.
CCBot has nine matches, not the count implied by three per day. PerplexityBot has thirteen matches, giving about 0.9 per nominal day rather than 1.9. Bytespider’s 615 of 805 requests are 76.4%; OAI-SearchBot’s 152 of 477 are 31.9%. These are request shares, not proof of successful rule downloads.
| Claimed identity | Requests | robots.txt matches | Matches / 14 |
|---|---|---|---|
| Bytespider | 805 | 615 | 43.9 |
| ClaudeBot | 10 227 745 | 601 | 42.9 |
| OAI-SearchBot | 477 | 152 | 10.9 |
| Applebot | 958 | 147 | 10.5 |
| Googlebot | 875 | 124 | 8.9 |
| bingbot | 4 357 | 69 | 4.9 |
| PerplexityBot | 607 | 13 | 0.9 |
| CCBot | 167 | 9 | 0.6 |
| Amazonbot | 10 700 | 9 | 0.6 |
| ChatGPT-User | 1 047 | 1 | 0.1 |
| meta-externalagent | 1 851 | 0 | 0.0 |
| GPTBot | 1 464 | 0 | 0.0 |
A mean is not an update deadline
Even if 615 requests covered exactly fourteen days, dividing elapsed time by the count would not reveal the longest gap, the next request or the moment a rule took effect. Multiple workers, several origins, clustered traffic and retries can produce the same average.
A zero count only means no match in this aggregation. It does not establish that the service never obtained rules. Shared caches, different agents, missing logs or failed attribution remain possible. Conversely, a fourteen-day-old cached copy is not automatically justified: RFC 9309 recommends refreshing within 24 hours unless robots.txt is unreachable. The earlier seven-minute effect claim lacks a timestamped before/after record in the reviewed package.
Crawler classes and errors matter
ChatGPT-User is documented as a user-triggered agent rather than an automatic web crawler; robots.txt may not apply to those requests. It should not be ranked as though it had the same scheduled refresh task as every crawler. Google also documents shared robots.txt caches across crawlers.
RFC 9309 allows access when robots.txt is unavailable, such as a 4xx response, and requires a disallow assumption for server/network failure, with a long-failure exception. Google has additional handling: ordinary 4xx responses mean no robots restrictions, but 429 is different. For server errors, Google initially pauses crawling for twelve hours, then can use the last good version; without one, crawling can resume unrestricted. A 503 is therefore not a dependable permanent block.
A 403 on robots.txt does not grant access through authentication or a firewall protecting other URLs. A 304 can validly revalidate a cached file. Serving a clear 200 with current text is useful, but insisting that every legitimate response must be 200 would be incorrect.
A better measurement
A useful follow-up records an explicit time interval and timezone, origin, exact path, method, status and rule version. Operator identity should be checked separately. Origin and CDN logs need consideration, and timestamps are required to calculate gaps. A content hash, a fingerprint calculated from the file contents, or a version identifier ties a successful fetch to the changed rules.
Google Search Console provides a robots.txt recrawl request for critical changes; that is not an immediate crawl guarantee for the affected pages. Other operators need their own current documentation. Urgent protection belongs in access controls or appropriate rate limits, with an explicit scope. robots.txt does not itself remove URLs from an index; readable noindex, removal of content or restricted access address different goals.
Sources
Related articles
