robots.txt for Search, Training and Assistant Crawlers: What Each Block Prevents

robots.txt expresses preferences for automated access. Search indexing, model training and user-requested retrieval are different purposes, but a provider can reuse a crawl for several permitted purposes. A user-agent name describes a claimed role; it does not authenticate a request.

A crawler is a program that retrieves web pages automatically. Indexing adds information to a search service’s catalogue; training uses examples to develop a model. User-requested retrieval fetches a page for a particular question. A product token is the name addressed by a robots.txt rule. The HTTP User-Agent is a name supplied in the request itself; the two are not always identical.

A useful policy starts with the desired result: retaining search visibility, declining specified training uses, or restricting direct retrieval. The relevant product tokens and enforcement methods differ.

Purposes and crawler identities

Googlebot and Bingbot crawl for search. A block can prevent fresh content from being processed and harm visibility, but does not guarantee removal from results within weeks. Known URLs can remain indexed. Restricting unhelpful crawl paths can also be intentional; a production-wide Disallow: / needs careful review.

GPTBot and ClaudeBot support model development. CCBot builds Common Crawl’s public dataset, which also serves purposes beyond training. Google-Extended is a usage-control token covering specified training and grounding uses, not a separate crawler. Grounding adds retrieved information to an answer. These services do not necessarily fetch every page or fetch only once.

OAI-SearchBot, PerplexityBot and Claude-SearchBot support search services. ChatGPT-User, Perplexity-User and Claude-User support user-requested retrieval. OpenAI says robots.txt rules may not apply to ChatGPT-User; Perplexity says its user fetcher generally ignores them. Anthropic states that its bots honor robots.txt. These are provider-specific behaviors.

What a block can and cannot change

Blocking a training-specific token need not block search. However, “no visibility consequences” is too broad: Google-Extended also controls grounding, and a block cannot guarantee that no copy reaches any training dataset through another source.

Google-Extended is not Googlebot

Google-Extended controls use of crawled content for future Gemini generations powering Gemini Apps and the Vertex AI API for Gemini, plus grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google states that this token does not affect inclusion or ranking in Google Search; that statement does not extend to every Google AI product.

OpenAI separates GPTBot training preferences from OAI-SearchBot search preferences. Opting out of OAI-SearchBot excludes the site from ChatGPT search answers according to OpenAI, although navigational links can remain. ChatGPT-User does not control search inclusion. An allowed crawl does not guarantee a citation or a visit.

Limits of retrospective control

New rules take effect after the service retrieves and processes the file. Cached copies can delay that change. A new disallow is not a command to erase previously collected archives or remove information from an already trained model.

Removal requests and access controls are separate measures. robots.txt alone provides no retrospective deletion guarantee, including for copies obtained through third parties.

llms.txt: content guidance

llms.txt offers a selected guide to useful content in Markdown, a text format with simple markers for headings and links. The proposal’s August 2026 revision also supports files below the site root, such as /docs/llms.txt, covering that path.

The proposal does not require every crawler to use the file and does not define access authorization. Listing a URL neither grants a licence nor overrides robots.txt, authentication or other access controls. The guide needs accurate links and maintenance; creating it does not guarantee better search placement.

Limits of robots.txt

robots.txt is a protocol for cooperating automated clients, not an access-control mechanism. Noncompliant clients and some user-triggered fetchers can still request public content. Sensitive content needs actual authorization checks.

A blocked URL can still appear in search results without a fetched page description. For supported search engines, noindex requests exclusion from indexing, but the crawler must be able to retrieve that directive. Combining a crawl block with a new noindex can prevent its discovery.

Details that change the result

Google-Extended is not an HTTP identity

Google-Extended has no separate HTTP user-agent string. Google uses existing crawler identities and interprets the token as a product-use preference. A server rule matching only “Google-Extended” therefore does not implement that preference. Blocking the real fetching crawler can also affect Search; it is not an equivalent substitute.

CCBot feeds a public archive

Common Crawl identifies its crawler as CCBot/2.0 and documents compliance with robots.txt. Data is available through Amazon S3 under Common Crawl’s terms. The dataset samples the web rather than guaranteeing a complete archive of every site. Blocking individual model-provider crawlers leaves this separate distribution route open unless separately addressed.

Duplicate groups combine their rules

A group begins with one or more User-agent lines and contains the rules for those names. Disallow limits retrieval of a path; Allow permits it. A path is the part of a URL after the site name, such as /blog/.

RFC 9309 combines groups matching the same product token, case-insensitively. Within the combined rules, the most specific matching path wins. Two GPTBot groups containing Disallow: / and Allow: /blog/ therefore permit /blog/. Duplicate groups should be reviewed together.