EnglishHow Often Crawlers Really Read robots.txt: Between Zero and 44 Times a Day
Contents
A line in robots.txt is not a switch. It is a file that has to be fetched again before anything changes, and until that happens the old rules keep applying.
How long “until that happens” lasts was measured on one server over fourteen days, across every crawler that identified itself. The answer runs from seven minutes to never.
Why the reading frequency decides the effect
A crawler does not read robots.txt before every request; that would double the traffic on both sides. It fetches the file, keeps the answer, and works from that copy for a while.
How long “a while” lasts is up to the crawler. A convention of roughly a day is widely followed, and nothing enforces it.
So a new rule has two separate starting points. It exists on the server the moment the file is saved. It exists for a given crawler only after that crawler’s next fetch. Between those two moments the crawler behaves exactly as before, and nothing on the server can tell it otherwise.
What was measured
Fourteen consecutive days of access logs, 22 August to 4 September 2026, from one small server, covering both of the hostnames it answers under.
Counted per crawler identity: every request, and separately every request for robots.txt. Twelve identities appeared in the period.
One detail of method deserves naming, because getting it wrong invents a result. The log writes a different prefix for each hostname. A first pass matched only one of them and produced a clean, confident table in which several crawlers had apparently never fetched the rules at all. The numbers below come from a pass that covers both.
From zero to forty-four times a day
Bytespider fetched the file 43.9 times a day. ClaudeBot 42.9 – roughly every thirty-four minutes.
Then a wide gap. OAI-SearchBot 10.9, Applebot 10.5, Googlebot 8.9, bingbot 4.9.
Then a long tail: CCBot 3.0, PerplexityBot 1.9, Amazonbot 0.6, ChatGPT-User 0.1.
And two that did not fetch it once in fourteen days.
The ranking is not the point. The point is that a single edit to a single file reaches the top of that list within the hour and the bottom of it on a timescale nobody on the server can name.
One reads the rules more often than the pages
Bytespider is the odd one. Of 805 requests in fourteen days, 615 were for robots.txt – three quarters of everything it asked for.
That leaves 190 actual pages in two weeks, against 615 checks of whether fetching them was allowed. Whatever the reason, the practical consequence is unusual: a rule aimed at that crawler takes effect almost at once, because it is asking almost constantly.
OAI-SearchBot shows a milder version of the same shape: 152 of 477 requests, just under a third.
Two never fetched it at all
GPTBot requested 1,464 pages in the period and the rules file zero times. meta-externalagent requested 1,851 pages and the rules file zero times.
That needs stating carefully. It is not evidence that anything was ignored: a copy fetched before the period would still be in force, and fourteen days is not forever. The measurement says only what it says – within this window, on this server, those two never asked.
What follows from it is about timing alone. A rule written for either of them has no knowable moment at which it starts to apply. It may have applied for weeks already, or it may apply next month.
What this means for a change
Three things, in the order they matter.
A change to robots.txt is not an event with a known completion. Its arrival is staggered across every crawler by their own habits, and the spread measured here covers three orders of magnitude.
Where a change is urgent – a crawler taking too much traffic – the frequency decides whether robots.txt is the right instrument at all. Against something that reads it every half hour it is excellent: one such change on this server took effect within seven minutes. Against something that has not read it in a fortnight, a block on the server is the only mechanism with a knowable start.
And the frequency is measurable on any server that keeps logs. One pass over them answers a question no external tool can: not whether a crawler obeys the rules, but when it would even notice that they had changed.
Questions and answers
Can a crawler be made to reread robots.txt sooner?
Only to a limited extent. RFC 9309 allows a crawler to cache the file under the usual HTTP caching rules and recommends not using a cached copy for more than 24 hours while the file is reachable. A crawler may honour a short Cache-Control value, but it is not obliged to. The measurement shows how far the habits diverge: from more than 40 fetches a day to none in fourteen days.
For Googlebot, Search Console offers a way to request a fresh fetch of robots.txt. For the other crawlers there is no such switch. Where a change has to take effect at once, a block on the server remains the only option.
What happens when robots.txt responds with an error?
Then the status code decides, and the two classes of error point in opposite directions. RFC 9309 distinguishes two cases:
- A code from 400 to 499 means that no rules exist. The crawler may then fetch everything, including areas the last version disallowed.
- A server error from 500 to 599, or a file that cannot be reached at all, means the opposite: the crawler is to assume a complete disallow. If the state persists for a long time, around 30 days, it may fall back on a cached copy or treat the file as absent.
According to its own documentation, Google treats a 429 like a server error rather than like the other 4xx codes.
In practice the risk therefore lies less in the file than in what sits in front of it. A firewall rule, a rate limit or a catch-all rule for retired paths that also matches /robots.txt changes the rules without anyone touching the file. A 403 opens the whole site to the crawler; a 503 closes it. The robots.txt file should therefore be exempt from such rules and always answer with 200, or with 404 only where there are deliberately no rules.
Does a Disallow rule remove a page from the search index?
No. The robots.txt file controls fetching, not indexing: a disallowed address can stay in the index, for instance because other pages link to it, only without its content. Taking a page out of the index requires noindex, and the crawler can only read that if it is allowed to fetch the page.
Related articles
