LW IT Solutions

robots.txt & llms.txt Generator for AI Crawlers

Save tool

Search crawlers, training crawlers and the crawlers that answer a live question are three different things, and blocking one of them has nothing to do with the others. This generator keeps them apart and says what each decision costs.

Site address

Site name

One sentence about the site, for llms.txt

Who is allowed in

Paths to keep out of every crawler, one per line

Articles that explain this

robots.txt for Search, Training and Assistant Crawlers: What Each Block Prevents Digital Marketing
Grouping Search Terms by Topic and Checking AI Visibility in Search Console Tutorials
SEO and GEO (Generative Engine Optimization): How AI Search Changes Optimisation Digital Marketing

Other tools

Related articles