GPTBot, ClaudeBot and Google-Extended: What an Entry in robots.txt Actually Does

A single file at the root of a website decides who may read it. It is plain text, it has no password, and everything in it is a request rather than a rule.

Since language models began learning from the open web, that file carries four new names. Adding them costs one line each. Whether the line does anything depends on details that are easy to get wrong – and on one case where the file grants everything precisely because it seems to be missing.

What robots.txt is, and what it is not

robots.txt is a text file at the root of a domain that lists which addresses a crawler may fetch. Any crawler reads it first, before anything else.

What it is not: a lock. Nothing enforces it. A crawler that ignores the file gets the pages anyway, and the only consequence is reputational. The large, named crawlers do respect it – that is the point of having a name at all, since an unnamed crawler cannot be addressed and cannot be seen to comply.

Which means the file works exactly to the extent that the other side chooses to be identifiable. That is worth stating plainly before anything is written into it.

The four names that currently matter

GPTBot fetches pages for OpenAI’s model training. It is separate from the crawler that fetches a page because someone asked a question about it – two different jobs, two different names.

ClaudeBot is the equivalent for Anthropic’s models.

CCBot belongs to Common Crawl, a public archive of the web. It is the oldest of the four and the most consequential, because its archive feeds many other projects – blocking one model provider while leaving CCBot open means the pages still reach model training, one step removed.

Google-Extended is the odd one out, and the next section is about why.

Each is addressed the same way: a group naming the crawler, then the paths it may not have. A group applies to exactly one name, and a crawler with its own group ignores the general group entirely – a rule worth knowing before writing anything, and one covered in detail in the piece on canonical, hreflang and noindex.

Google-Extended does not do what its name suggests

This is the one that gets misread, and the misreading is expensive in both directions.

Google-Extended is not a crawler. No request ever arrives carrying that name. It is a label used to answer a separate question: may the content Google already fetched be used for training its models.

The fetching itself is done by Googlebot, under its own name and its own rules. So blocking Google-Extended changes nothing about how a site is crawled and nothing about whether it appears in search results. The pages are still fetched, still indexed, still ranked.

The reverse mistake costs more. Blocking Googlebot to keep content out of AI training removes the site from search results as well – the one crawler doing both jobs is the one whose name was blocked. What was meant as a decision about training becomes a decision about visibility.

Two names, two questions. Getting them the wrong way round is the single most damaging thing possible in this file.

What thirteen real files actually say

Reading is more useful than guessing. Fifteen large sites were checked on 1 September 2026; thirteen returned a readable file.

GPTBot is named in six of them, ClaudeBot in six, CCBot in five, Google-Extended in four. So roughly half of these sites have taken a position, and the other half have not – which is itself a position, since silence means permission.

The files differ enormously in size. One runs to over 33,000 characters with thirteen groups; another has a single group and 62 characters. One large retailer names ClaudeBot twice, in two separate groups, both with the same rule. Duplicate groups for one name are supposed to be merged, and here it makes no difference because both say the same thing – but a duplicate with two different rules is a coin toss, and it is worth checking for.

Two of the fifteen did not return a file at all. That case turns out to be the most interesting one.

A file that does not answer grants everything

The rules for what happens when robots.txt cannot be read are written down, and they run in opposite directions depending on the reason.

If the file answers with a client error – not found, gone, forbidden, anything in the 400s – a crawler may treat the site as fully open. No file, no restrictions. This is sensible: most sites have no robots.txt, and requiring one before crawling anything would break the web.

If the server fails instead – a 500, a timeout – the crawler should treat the site as fully closed, at least for a while. Also sensible: a broken server is not a permission slip, and a rule set that vanished for an hour should not open a site for that hour.

So the same absence means opposite things depending on which side broke. And a site that answers its robots.txt with a bot defence – blocking unusual clients with a 403 or a 418 – has, by that very defence, told every rule-following crawler that everything is allowed. Two of the fifteen checked sites answer exactly like that.

The rules to write down are worth nothing if the file carrying them is not delivered.

What can be checked in a minute

Four questions settle the state of any domain.

Does the file answer with a plain 200, and is it actually text rather than an error page. Which of the four names appear, and does each have its own group. Is Googlebot treated separately from Google-Extended, so that the training decision has not accidentally become a visibility decision. And does any name appear twice with differing rules.

All of it is public, for any domain, and reading it takes one request. The indexing inspector reads robots.txt with the correct group precedence and reports which group actually decided.

The answer worth having is one sentence: whether this file says what its author believes it says – and whether it is being delivered at all.