robots.txt for Search, Training and Assistant Crawlers: What Each Block Prevents

One file, three audiences that have nothing to do with each other. A search crawler fetches a page so it can be found; a training crawler fetches it so a model can be built from it; an assistant fetches it because someone asked a question a minute ago. The three are told apart only by the name they announce themselves with, and blocking one says nothing about the others.

Which is why the usual instruction – block the AI crawlers – is not a decision but three decisions, and two of them cost something that is rarely stated when the line is added.

Three Kinds That Do Different Work

Search crawlers – Googlebot, Bingbot and the rest – build the index that ordinary results come from. Blocking them removes the site from search within weeks, which is why nobody does it on purpose and why it happens by accident: a staging Disallow: / that survives the move to production.

Training crawlers – GPTBot, Google-Extended, ClaudeBot, CCBot and others – collect material that models are built from. They fetch once and read everything; nothing they take comes back as a visit.

Assistant crawlers – OAI-SearchBot, ChatGPT-User, PerplexityBot – fetch a page because a question is being answered right now. What they take often comes back: as a citation, as a link, as the sentence someone reads instead of the search result.

What Each Block Prevents and Costs

The middle row is the one that can be decided on principle without paying for it. The bottom row is the one that gets decided by accident, because the crawler names sit next to the training ones in every list that circulates.

Google-Extended Is Not Googlebot

The distinction that causes the most hesitation: Google-Extended governs whether content is used for training Gemini and the Vertex models and for grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. It has no effect on crawling for Search, on indexing, or on ranking. Blocking it changes nothing about how a page performs in Google, and that is not a claim requiring trust – it is what the token exists for.

The same separation runs through the other providers. GPTBot is training; OAI-SearchBot feeds the search index behind the assistant; ChatGPT-User is the fetch that happens when a user asks. Blocking all three because the names look similar is the most common way a site quietly leaves the answers it wanted to be in.

Nothing Works Backwards

A block applies from the moment it is read. Whatever has already been fetched sits in corpora that were built from it, and those do not consult robots.txt again. A site that has been online for years and adds the lines today has decided about future crawls, not past ones.

That is worth saying plainly because the decision is often framed as removal. It is not removal; it is a fence around what has not been taken yet.

llms.txt: a Map, Not a Rule

The file at /llms.txt is the opposite of robots.txt in intent: it does not forbid, it points. A short markdown document naming what the site is and which pages carry the substance – a table of contents for something that reads without navigation.

Two things to be clear about. It is a proposal rather than a standard, and no crawler is obliged to look for it. And it does not restrict anything: a page listed there is not thereby permitted, and a page missing from it is not thereby protected. It costs ten minutes to write and is worth exactly what a good table of contents is worth – nothing at all if the pages behind it are thin.

What robots.txt Cannot Do

It is a request, not a lock. Every crawler decides for itself whether to obey, and the ones that do are the ones that publish a name to be blocked by. Anything that ignores the file is not affected by anything in it.

And it does not keep a page out of search results. A blocked page that others link to can still appear, without a description, because the crawler was not allowed to read what it says. Keeping something out of the index takes a noindex – which requires the crawler to be allowed in to read it – and keeping something private takes authentication. robots.txt sits below both, and works well for exactly what it is: traffic direction for the crawlers that agreed to be directed.

Three Details Behind the Names

Google-Extended cannot be blocked at the server

Google-Extended has no user agent string of its own. Google’s crawler documentation states that crawling is done with the existing Google user agent strings and that the token is used in robots.txt purely in a control capacity. No request ever arrives under that name, so a firewall or server rule matching it catches nothing, and a rule against the agent that actually fetches, Googlebot, takes the site out of Search. For Google’s training and grounding question, the robots.txt entry is the only lever.

CCBot feeds a public archive

CCBot is the crawler of Common Crawl. It identifies itself as CCBot/2.0 and follows robots.txt. The crawl data is stored on Amazon S3 and published as an open dataset with as few restrictions as possible. What CCBot fetches therefore does not stay with a single model provider; it is available to anyone who downloads the archive. Blocking the training crawlers of individual providers while CCBot stays open leaves that indirect route in place.

A name listed twice is merged, not raffled

Files that grow over years sometimes name the same crawler in two groups. RFC 9309 settles that case: crawlers match the name case-insensitively, and if more than one group matches, their rules are combined into a single group. Within that merged group the longest matching rule decides, as in any other group. Two groups for GPTBot, one with Disallow: / and the other with Allow: /blog/, therefore leave /blog/ open, which may not be what was intended when the blocking group was added. Searching the file for each crawler name shows such duplicates in seconds.