{"id":13590,"date":"2026-10-19T07:35:00","date_gmt":"2026-10-19T05:35:00","guid":{"rendered":"https:\/\/www.lukaswojcik.com\/blog\/?p=13590"},"modified":"2026-09-25T09:45:53","modified_gmt":"2026-09-25T07:45:53","slug":"410-gone-on-robots-txt-how-a-4xx-answer-on-the-rules-file-affects-crawler","status":"publish","type":"post","link":"https:\/\/www.lukaswojcik.com\/blog\/en\/cloud-ai\/tutorials-en-cloud-ai\/410-gone-on-robots-txt-how-a-4xx-answer-on-the-rules-file-affects-crawler\/","title":{"rendered":"410 Gone on robots.txt: How a 4xx Answer on the Rules File Affects Crawler Behaviour"},"content":{"rendered":"

A refusal is meant to end a conversation. The clearest one the web has is 410 Gone<\/code>: not “try later”, not “moved”, but “this existed and it does not any more”.<\/p>\n

A small server sent that refusal 732,498 times in a single day. The crawler receiving most of them did not slow down, and had not slowed down for eleven days.<\/p>\n

\n\"Four
Four steps, each correct on its own, and together they kept a crawler busy for eleven days.<\/figcaption><\/figure>\n

What a 410 actually says<\/h2>\n

Several answers cover the ways a page can be unavailable, and they say quite different things.<\/p>\n

404 Not Found<\/code> means nothing is here. It leaves open whether something was here before, or might be here tomorrow. A crawler treats it as temporary and comes back.<\/p>\n

410 Gone<\/code> means something was here, it was removed on purpose, and there is no forwarding address. It is the strongest way to say a page is finished, and search engines drop such addresses faster than they drop a 404.<\/p>\n

So answering thousands of dead addresses with 410 is the correct choice. The mistake was somewhere else entirely.<\/p>\n

The rule caught the one file that could have stopped it<\/h2>\n

The refusal was implemented as a single rule: every address under the retired hostnames answers 410. Two exceptions were carved out, both deliberate and both narrow.<\/p>\n

robots.txt<\/code> was not among them.<\/p>\n

That is easy to miss, because robots.txt is not a page. It is the file about the pages. Under a rule saying “everything here is gone” it is simply an address like any other, and it answered 410 like any other.<\/p>\n

Why a 4xx answer means no restrictions<\/h2>\n

This is the part that turns an oversight into an invitation.<\/p>\n

The standard is explicit about what happens when robots.txt cannot be fetched. A 5xx answer – the server is broken – means a crawler should treat the whole site as forbidden and come back later. A 4xx answer means the opposite: there are no rules on this host, and everything may be crawled.<\/p>\n

The reasoning behind that is sound. A missing robots.txt has always meant “no restrictions”, and nothing distinguishes “this host never had rules” from “this host removed them”. 404 and 410 land in the same category.<\/p>\n

So the refusal arrived exactly backwards. It did not say stay away. It said, in the one sentence a crawler reads before deciding anything, that nothing here was off limits.<\/p>\n

\n\"A
\n<\/figure>\n

An address space with no bottom<\/h2>\n

A wrong answer only becomes expensive when there is somewhere to spend it.<\/p>\n

The addresses being requested carried a parameter encoding where the crawler had come from. Every page therefore linked onward to addresses that had never existed before, and those linked onward again. The supply never ran out.<\/p>\n

The numbers show it plainly. Of 698,400 requests in one day, 696,907 were for different addresses – two requests in a thousand were repeats. A crawler working through such a space is not doing anything wrong; it is following links, which is its entire job, and it will never finish.<\/p>\n

What a single refusal weighs<\/h2>\n

The last surprise was in the size of it.<\/p>\n

The message itself is small: 415 bytes of plain text saying the resource is gone. The headers wrapped around it came to 1,949 bytes, and roughly 1,700 of those were a content security policy.<\/p>\n

That policy exists to restrict what a page is allowed to load. On a refusal there is no page – nothing to load, nothing to restrict, nothing the header can protect. It was attached to every one of the 732,498 answers all the same, because it had been set to apply always, which is exactly the right setting for real pages.<\/p>\n

The result: 2,364 bytes to say no, and 1.65 GB in a day spent saying it.<\/p>\n

What follows from this<\/h2>\n

Three things, in order of effect.<\/p>\n

A rule that makes a host disappear needs an exception for the file describing the host. robots.txt, and anything else a crawler reads in order to decide, has to stay readable even when everything it describes is gone. A reachable robots.txt carrying a single Disallow: \/<\/code> ends the traffic at its source rather than absorbing it.<\/p>\n

Headers meant for pages do not belong on refusals. Restricting them to the hosts that actually serve pages cut each answer from 2,364 bytes to 850 – the same refusal at roughly a third of the weight.<\/p>\n

And a parameter that generates fresh addresses turns any crawl into an endless one. Where such a space sits behind removed content, removing the content is not the end of the work.<\/p>\n

\n

Questions and answers<\/h2>\n

Does a Disallow: \/ also remove the old addresses from the search index?<\/h3>\n

Not directly. A Disallow forbids fetching, not indexing. A blocked address is no longer requested, so the crawler no longer sees its 410 either. Addresses that are already known, or linked from elsewhere, can therefore stay in the index as a bare address without content. A noindex does not help here, because it too would sit on a page that is no longer fetched.<\/p>\n

Anyone who wants both has to choose an order: first the important addresses need a chance to deliver their 410, then the block ends the traffic. With an address space that has no bottom, like the one in the article, that plan does not work, because the refusals would never be finished. There the block is the better choice, even if individual addresses stay in the index for a while.<\/p>\n

Why not answer robots.txt with a 503 so that the whole site counts as blocked?<\/h3>\n

Because that block only lasts for a while. RFC 9309 treats a server error on robots.txt as a complete disallow at first, but allows a crawler, once the file has been unreachable for a long time (the standard gives 30 days as an example), to act as if there were no robots.txt, or to keep using a cached copy. In the first case the host is back to having no rules, just as with the 410. A reachable file containing Disallow: \/ is the form that holds permanently.<\/p>\n

How quickly does a corrected robots.txt take effect?<\/h3>\n

With a delay of up to a day. RFC 9309 says a crawler should not use a cached robots.txt for more than 24 hours; until then it may still act on the old answer, which here meant behaving as if there were no rules. Whether the fix works therefore shows in the following day’s traffic, not in the first hour.<\/p>\n<\/div>\n

\n

Sources<\/h2>\n