GPTBot, ClaudeBot and Google-Extended: What an Entry in robots.txt Actually Does
Four crawler names decide whether a site is read for AI training. Naming them costs one line each – but a file that answers with an error grants everything, and one of the four does not do what its reputation suggests.
