Three Kinds of Crawler, One robots.txt
Contents
One file, three audiences that have nothing to do with each other. A search crawler fetches a page so it can be found; a training crawler fetches it so a model can be built from it; an assistant fetches it because someone asked a question a minute ago. The three are told apart only by the name they announce themselves with, and blocking one says nothing about the others.
Which is why the usual instruction – block the AI crawlers – is not a decision but three decisions, and two of them cost something that is rarely stated when the line is added.
Three Kinds That Do Different Work
Search crawlers – Googlebot, Bingbot and the rest – build the index that ordinary results come from. Blocking them removes the site from search within weeks, which is why nobody does it on purpose and why it happens by accident: a staging Disallow: / that survives the move to production.
Training crawlers – GPTBot, Google-Extended, ClaudeBot, CCBot and others – collect material that models are built from. They fetch once and read everything; nothing they take comes back as a visit.
Assistant crawlers – OAI-SearchBot, ChatGPT-User, PerplexityBot – fetch a page because a question is being answered right now. What they take often comes back: as a citation, as a link, as the sentence someone reads instead of the search result.
What Each Block Prevents and Costs
| Blocking | Prevents | Costs |
|---|---|---|
| Search | Nothing worth preventing. | The site disappears from Google and Bing. |
| Training | That new material enters the next model. | Nothing measurable – not ranking, not visibility. |
| Assistants | That the site is quoted in an answer. | Exactly that visibility – the citation and the link that comes with it. |
The middle row is the one that can be decided on principle without paying for it. The bottom row is the one that gets decided by accident, because the crawler names sit next to the training ones in every list that circulates.
Google-Extended Is Not Googlebot
The distinction that causes the most hesitation: Google-Extended governs whether content is used for training Gemini and the Vertex models. It has no effect on crawling for Search, on indexing, or on ranking. Blocking it changes nothing about how a page performs in Google, and that is not a claim requiring trust – it is what the token exists for.
The same separation runs through the other providers. GPTBot is training; OAI-SearchBot feeds the search index behind the assistant; ChatGPT-User is the fetch that happens when a user asks. Blocking all three because the names look similar is the most common way a site quietly leaves the answers it wanted to be in.
Nothing Works Backwards
A block applies from the moment it is read. Whatever has already been fetched sits in corpora that were built from it, and those do not consult robots.txt again. A site that has been online for years and adds the lines today has decided about future crawls, not past ones.
That is worth saying plainly because the decision is often framed as removal. It is not removal; it is a fence around what has not been taken yet.
llms.txt: a Map, Not a Rule
The file at /llms.txt is the opposite of robots.txt in intent: it does not forbid, it points. A short markdown document naming what the site is and which pages carry the substance – a table of contents for something that reads without navigation.
Two things to be clear about. It is a proposal rather than a standard, and no crawler is obliged to look for it. And it does not restrict anything: a page listed there is not thereby permitted, and a page missing from it is not thereby protected. It costs ten minutes to write and is worth exactly what a good table of contents is worth – nothing at all if the pages behind it are thin.
What robots.txt Cannot Do
It is a request, not a lock. Every crawler decides for itself whether to obey, and the ones that do are the ones that publish a name to be blocked by. Anything that ignores the file is not affected by anything in it.
And it does not keep a page out of search results. A blocked page that others link to can still appear, without a description, because the crawler was not allowed to read what it says. Keeping something out of the index takes a noindex – which requires the crawler to be allowed in to read it – and keeping something private takes authentication. robots.txt sits below both, and works well for exactly what it is: traffic direction for the crawlers that agreed to be directed.
Related articles
