{"id":14728,"date":"2026-10-25T07:35:00","date_gmt":"2026-10-25T06:35:00","guid":{"rendered":"https:\/\/www.lukaswojcik.com\/blog\/?p=14728"},"modified":"2026-09-25T09:45:54","modified_gmt":"2026-09-25T07:45:54","slug":"how-often-crawlers-really-read-robots-txt-between-zero-and-44-times-a-day","status":"publish","type":"post","link":"https:\/\/www.lukaswojcik.com\/blog\/en\/cloud-ai\/tutorials-en-cloud-ai\/how-often-crawlers-really-read-robots-txt-between-zero-and-44-times-a-day\/","title":{"rendered":"How Often Crawlers Really Read robots.txt: Between Zero and 44 Times a Day"},"content":{"rendered":"

A line in robots.txt is not a switch. It is a file that has to be fetched again before anything changes, and until that happens the old rules keep applying.<\/p>\n

How long “until that happens” lasts was measured on one server over fourteen days, across every crawler that identified itself. The answer runs from seven minutes to never.<\/p>\n

\n\"Eight
Fourteen days, twelve identities, counted from one server’s own access logs.<\/figcaption><\/figure>\n

Why the reading frequency decides the effect<\/h2>\n

A crawler does not read robots.txt before every request; that would double the traffic on both sides. It fetches the file, keeps the answer, and works from that copy for a while.<\/p>\n

How long “a while” lasts is up to the crawler. A convention of roughly a day is widely followed, and nothing enforces it.<\/p>\n

So a new rule has two separate starting points. It exists on the server the moment the file is saved. It exists for a given crawler only after that crawler’s next fetch. Between those two moments the crawler behaves exactly as before, and nothing on the server can tell it otherwise.<\/p>\n

What was measured<\/h2>\n

Fourteen consecutive days of access logs, 22 August to 4 September 2026, from one small server, covering both of the hostnames it answers under.<\/p>\n

Counted per crawler identity: every request, and separately every request for robots.txt<\/code>. Twelve identities appeared in the period.<\/p>\n

One detail of method deserves naming, because getting it wrong invents a result. The log writes a different prefix for each hostname. A first pass matched only one of them and produced a clean, confident table in which several crawlers had apparently never fetched the rules at all. The numbers below come from a pass that covers both.<\/p>\n

From zero to forty-four times a day<\/h2>\n

Bytespider fetched the file 43.9 times a day. ClaudeBot 42.9 – roughly every thirty-four minutes.<\/p>\n

Then a wide gap. OAI-SearchBot 10.9, Applebot 10.5, Googlebot 8.9, bingbot 4.9.<\/p>\n

Then a long tail: CCBot 3.0, PerplexityBot 1.9, Amazonbot 0.6, ChatGPT-User 0.1.<\/p>\n

And two that did not fetch it once in fourteen days.<\/p>\n

The ranking is not the point. The point is that a single edit to a single file reaches the top of that list within the hour and the bottom of it on a timescale nobody on the server can name.<\/p>\n

\n\"One
\n<\/figure>\n

One reads the rules more often than the pages<\/h2>\n

Bytespider is the odd one. Of 805 requests in fourteen days, 615 were for robots.txt – three quarters of everything it asked for.<\/p>\n

That leaves 190 actual pages in two weeks, against 615 checks of whether fetching them was allowed. Whatever the reason, the practical consequence is unusual: a rule aimed at that crawler takes effect almost at once, because it is asking almost constantly.<\/p>\n

OAI-SearchBot shows a milder version of the same shape: 152 of 477 requests, just under a third.<\/p>\n

Two never fetched it at all<\/h2>\n

GPTBot requested 1,464 pages in the period and the rules file zero times. meta-externalagent requested 1,851 pages and the rules file zero times.<\/p>\n

That needs stating carefully. It is not evidence that anything was ignored: a copy fetched before the period would still be in force, and fourteen days is not forever. The measurement says only what it says – within this window, on this server, those two never asked.<\/p>\n

What follows from it is about timing alone. A rule written for either of them has no knowable moment at which it starts to apply. It may have applied for weeks already, or it may apply next month.<\/p>\n

What this means for a change<\/h2>\n

Three things, in the order they matter.<\/p>\n

A change to robots.txt is not an event with a known completion. Its arrival is staggered across every crawler by their own habits, and the spread measured here covers three orders of magnitude.<\/p>\n

Where a change is urgent – a crawler taking too much traffic – the frequency decides whether robots.txt is the right instrument at all. Against something that reads it every half hour it is excellent: one such change on this server took effect within seven minutes. Against something that has not read it in a fortnight, a block on the server is the only mechanism with a knowable start.<\/p>\n

And the frequency is measurable on any server that keeps logs. One pass over them answers a question no external tool can: not whether a crawler obeys the rules, but when it would even notice that they had changed.<\/p>\n

\n

Questions and answers<\/h2>\n

Can a crawler be made to reread robots.txt sooner?<\/h3>\n

Only to a limited extent. RFC 9309 allows a crawler to cache the file under the usual HTTP caching rules and recommends not using a cached copy for more than 24 hours while the file is reachable. A crawler may honour a short Cache-Control<\/code> value, but it is not obliged to. The measurement shows how far the habits diverge: from more than 40 fetches a day to none in fourteen days.<\/p>\n

For Googlebot, Search Console offers a way to request a fresh fetch of robots.txt. For the other crawlers there is no such switch. Where a change has to take effect at once, a block on the server remains the only option.<\/p>\n

What happens when robots.txt responds with an error?<\/h3>\n

Then the status code decides, and the two classes of error point in opposite directions. RFC 9309 distinguishes two cases:<\/p>\n