{"id":10253,"date":"2026-08-15T00:42:16","date_gmt":"2026-08-14T22:42:16","guid":{"rendered":"https:\/\/www.lukaswojcik.com\/blog\/toolbox\/robots-llms-txt-ai-crawler-generator\/"},"modified":"2026-08-15T00:42:16","modified_gmt":"2026-08-14T22:42:16","slug":"robots-llms-txt-ai-crawler-generator","status":"publish","type":"page","link":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-llms-txt-ai-crawler-generator\/","title":{"rendered":"robots.txt &#038; llms.txt Generator for AI Crawlers"},"content":{"rendered":"<div class=\"gtm-analyser-container\" style=\"background: var(--bg-panel, #1e1e24); padding: 25px; border-radius: 8px; border: 1px solid var(--border, #2a2a35);\">\n<p style=\"color: var(--text-secondary, #a0a0b0); margin-bottom: 20px;\">Search crawlers, training crawlers and the crawlers that answer a live question are three different things, and blocking one of them has nothing to do with the others. This generator keeps them apart and says what each decision costs.<\/p>\n<div style=\"display: flex; gap: 12px; flex-wrap: wrap;\">\n<div style=\"flex: 2; min-width: 240px;\">\n            <label for=\"rb-host\" style=\"color: var(--text-secondary, #a0a0b0); display: block; font-size: 0.85rem; margin-bottom: 5px;\">Site address<\/label><br \/>\n            <input type=\"text\" id=\"rb-host\" class=\"form-control\" style=\"width: 100%; padding: 10px; background: var(--bg-body, #14141a); border: 1px solid var(--border, #2a2a35); color: var(--text-primary, #e8e8ee); border-radius: 6px; font-family: monospace; box-sizing: border-box;\" value=\"https:\/\/www.example.com\">\n        <\/div>\n<div style=\"flex: 2; min-width: 240px;\">\n            <label for=\"rb-name\" style=\"color: var(--text-secondary, #a0a0b0); display: block; font-size: 0.85rem; margin-bottom: 5px;\">Site name<\/label><br \/>\n            <input type=\"text\" id=\"rb-name\" class=\"form-control\" style=\"width: 100%; padding: 10px; background: var(--bg-body, #14141a); border: 1px solid var(--border, #2a2a35); color: var(--text-primary, #e8e8ee); border-radius: 6px; font-family: monospace; box-sizing: border-box;\" value=\"Example\">\n        <\/div>\n<\/p><\/div>\n<div style=\"margin-top: 12px;\">\n        <label for=\"rb-zusammenfassung\" style=\"color: var(--text-secondary, #a0a0b0); display: block; font-size: 0.85rem; margin-bottom: 5px;\">One sentence about the site, for llms.txt<\/label><br \/>\n        <input type=\"text\" id=\"rb-zusammenfassung\" class=\"form-control\" style=\"width: 100%; padding: 10px; background: var(--bg-body, #14141a); border: 1px solid var(--border, #2a2a35); color: var(--text-primary, #e8e8ee); border-radius: 6px; font-family: monospace; box-sizing: border-box;\" placeholder=\"Technical notes on analytics, networks and self-hosted infrastructure\">\n    <\/div>\n<div style=\"display: flex; gap: 14px; flex-wrap: wrap; margin-top: 15px;\">\n<div style=\"flex: 1; min-width: 240px;\">\n<div style=\"color: var(--text-secondary, #a0a0b0); font-size: 0.85rem; margin-bottom: 8px;\">Who is allowed in<\/div>\n<div id=\"rb-gruppen\"><\/div>\n<\/p><\/div>\n<div style=\"flex: 1; min-width: 240px;\">\n            <label for=\"rb-pfade\" style=\"color: var(--text-secondary, #a0a0b0); display: block; font-size: 0.85rem; margin-bottom: 5px;\">Paths to keep out of every crawler, one per line<\/label><br \/>\n            <textarea id=\"rb-pfade\" class=\"form-control\" style=\"width: 100%; height: 120px; padding: 10px; background: var(--bg-body, #14141a); border: 1px solid var(--border, #2a2a35); color: var(--text-primary, #e8e8ee); border-radius: 6px; font-family: monospace; font-size: 0.85rem; box-sizing: border-box;\">\/wp-admin\/<br \/>\n\/*?s=<br \/>\n\/danke\/<\/textarea>\n        <\/div>\n<\/p><\/div>\n<div style=\"display: flex; gap: 10px; flex-wrap: wrap; margin: 18px 0 0;\">\n        <button id=\"rb-btn\" class=\"button\" style=\"background: var(--accent, #7ee787); color: #0b1114; border: none; padding: 12px 24px; border-radius: 6px; font-weight: 700; cursor: pointer;\">Generate files<\/button>\n    <\/div>\n<div id=\"rb-ergebnis\" style=\"margin-top: 22px;\"><\/div>\n<\/div>\n<p><script>\n(function () {\n    'use strict';<\/p>\n<p>    const T = {\"einleitung\":\"Search crawlers, training crawlers and the crawlers that answer a live question are three different things, and blocking one of them has nothing to do with the others. This generator keeps them apart and says what each decision costs.\",\"label_host\":\"Site address\",\"label_name\":\"Site name\",\"label_zusammenfassung\":\"One sentence about the site, for llms.txt\",\"ph_zusammenfassung\":\"Technical notes on analytics, networks and self-hosted infrastructure\",\"label_gruppen\":\"Who is allowed in\",\"label_pfade\":\"Paths to keep out of every crawler, one per line\",\"knopf\":\"Generate files\",\"fehler_host\":\"The site address is missing.\",\"fehler_host_form\":\"The address has to be a bare origin such as https:\\\/\\\/www.example.com, without a path.\",\"fehler_pfad\":\"A path has to start with a slash: %s\",\"kopf_hinweis\":\"Generated with the robots.txt generator on lukaswojcik.com\",\"gruppe_suche\":\"Search engines\",\"gruppe_suche_text\":\"Googlebot, Bingbot and the rest of classic search. Blocking these removes the site from search results.\",\"gruppe_training\":\"AI training\",\"gruppe_training_text\":\"GPTBot, Google-Extended, ClaudeBot and others that collect material for training. Blocking these has no effect on search rankings.\",\"gruppe_assistent\":\"AI assistants\",\"gruppe_assistent_text\":\"OAI-SearchBot, PerplexityBot and the crawlers that fetch a page to answer a question right now. Blocking these keeps the site out of AI answers.\",\"gruppe_archiv\":\"Archives and data collectors\",\"gruppe_archiv_text\":\"CCBot, Diffbot and similar, whose corpora feed many downstream models at once.\",\"h_robots\":\"robots.txt\",\"h_llms\":\"llms.txt\",\"h_kennungen\":\"What each crawler gets\",\"h_befunde\":\"What that means\",\"llms_platzhalter\":\"One sentence on what this site is about.\",\"llms_wichtig\":\"Core pages\",\"llms_optional\":\"Additional\",\"llms_start\":\"Home\",\"llms_start_text\":\"entry point and overview\",\"llms_blog\":\"Blog\",\"llms_blog_text\":\"articles and tutorials\",\"llms_ueber\":\"About\",\"llms_ueber_text\":\"who is behind the site\",\"llms_sitemap\":\"Sitemap\",\"b_suche_gesperrt\":\"Search engines are blocked. That removes the site from Google and Bing within a few weeks.\",\"b_training_ohne_assistent\":\"Training is blocked while assistants stay allowed: the site can still be quoted in an answer, it just does not end up in the next model.\",\"b_google_extended\":\"Google-Extended only governs training for Gemini and Vertex AI. Blocking it changes nothing about ranking in Google Search.\",\"b_rueckwirkend\":\"A block only works from now on. Whatever has already been crawled stays in the corpora that were built from it.\",\"b_assistent_gesperrt\":\"AI assistants are blocked, so the site cannot appear as a source in their answers - which is exactly the visibility that generative engines hand out.\",\"b_alles_gesperrt\":\"One of the paths is a bare slash, which locks every crawler out of the whole site.\",\"b_platzhalter\":\"The paths use * or $. Google and Bing understand both, many smaller crawlers do not and read the line literally.\",\"b_kein_schutz\":\"robots.txt is a request, not a lock. A page kept out this way can still show up in search if others link to it - that takes a noindex header or an actual access control.\",\"b_freiwillig\":\"Every crawler decides for itself whether to obey. The list above covers those that publish a token and honour it.\"};<\/p>\n<p>    \/\/ --- reine Rechenlogik ---<\/p>\n<p>    \/\/ Die Kennungen stammen aus den Angaben der jeweiligen Betreiber. Wer\n    \/\/ eine Kennung sperrt, sperrt genau diesen Zugriff und nicht die Firma.\n    const BOTS = [\n        { name: 'Googlebot', gruppe: 'suche' },\n        { name: 'Googlebot-Image', gruppe: 'suche' },\n        { name: 'Bingbot', gruppe: 'suche' },\n        { name: 'DuckDuckBot', gruppe: 'suche' },\n        { name: 'Applebot', gruppe: 'suche' },\n        { name: 'YandexBot', gruppe: 'suche' },\n        { name: 'Seznam', gruppe: 'suche' },<\/p>\n<p>        { name: 'GPTBot', gruppe: 'training' },\n        { name: 'Google-Extended', gruppe: 'training' },\n        { name: 'ClaudeBot', gruppe: 'training' },\n        { name: 'anthropic-ai', gruppe: 'training' },\n        { name: 'Applebot-Extended', gruppe: 'training' },\n        { name: 'meta-externalagent', gruppe: 'training' },\n        { name: 'Bytespider', gruppe: 'training' },\n        { name: 'Amazonbot', gruppe: 'training' },\n        { name: 'cohere-ai', gruppe: 'training' },<\/p>\n<p>        { name: 'OAI-SearchBot', gruppe: 'assistent' },\n        { name: 'ChatGPT-User', gruppe: 'assistent' },\n        { name: 'PerplexityBot', gruppe: 'assistent' },\n        { name: 'Perplexity-User', gruppe: 'assistent' },\n        { name: 'YouBot', gruppe: 'assistent' },<\/p>\n<p>        { name: 'CCBot', gruppe: 'archiv' },\n        { name: 'Diffbot', gruppe: 'archiv' },\n        { name: 'Omgilibot', gruppe: 'archiv' },\n        { name: 'ImagesiftBot', gruppe: 'archiv' }\n    ];<\/p>\n<p>    const GRUPPEN = ['suche', 'training', 'assistent', 'archiv'];<\/p>\n<p>    function pfadeLesen(text) {\n        return String(text === undefined || text === null ? '' : text)\n            .split(\/\\r?\\n\/)\n            .map(function (s) { return s.trim(); })\n            .filter(function (s) { return s !== ''; });\n    }<\/p>\n<p>    function bauen(eingabe) {\n        const ergebnis = { ok: false, fehler: '', robots: '', llms: '', befunde: [], bots: [] };\n        const e = eingabe || {};<\/p>\n<p>        const host = String(e.host === undefined ? '' : e.host).trim().replace(\/\\\/+$\/, '');\n        if (host === '') { ergebnis.fehler = T.fehler_host; return ergebnis; }\n        if (!\/^https?:\\\/\\\/[^\\\/\\s]+$\/i.test(host)) { ergebnis.fehler = T.fehler_host_form; return ergebnis; }<\/p>\n<p>        const name = String(e.name === undefined ? '' : e.name).trim() || host.replace(\/^https?:\\\/\\\/\/i, '');\n        const pfade = pfadeLesen(e.pfade);\n        const erlaubt = {};\n        GRUPPEN.forEach(function (g) {\n            erlaubt[g] = e[g] === undefined ? (g === 'suche') : !!e[g];\n        });<\/p>\n<p>        \/\/ Ungueltige Pfade abfangen, bevor sie in der Datei stehen.\n        for (let i = 0; i < pfade.length; i++) {\n            if (pfade[i].charAt(0) !== '\/') {\n                ergebnis.fehler = T.fehler_pfad.replace('%s', pfade[i]);\n                return ergebnis;\n            }\n        }\n\n        const zeilen = [];\n        zeilen.push('# robots.txt fuer ' + host);\n        zeilen.push('# ' + T.kopf_hinweis);\n        zeilen.push('');\n\n        \/\/ Erst die allgemeine Gruppe, danach die einzelnen Kennungen.\n        zeilen.push('User-agent: *');\n        if (pfade.length) {\n            pfade.forEach(function (p) { zeilen.push('Disallow: ' + p); });\n        } else {\n            zeilen.push('Disallow:');\n        }\n        zeilen.push('');\n\n        const gesperrt = [];\n        GRUPPEN.forEach(function (gruppe) {\n            const botsDerGruppe = BOTS.filter(function (b) { return b.gruppe === gruppe; });\n            if (erlaubt[gruppe]) {\n                botsDerGruppe.forEach(function (b) {\n                    ergebnis.bots.push({ name: b.name, gruppe: gruppe, erlaubt: true });\n                });\n                return;\n            }\n            botsDerGruppe.forEach(function (b) {\n                ergebnis.bots.push({ name: b.name, gruppe: gruppe, erlaubt: false });\n                gesperrt.push(b.name);\n            });\n            zeilen.push('# ' + T['gruppe_' + gruppe]);\n            botsDerGruppe.forEach(function (b) { zeilen.push('User-agent: ' + b.name); });\n            zeilen.push('Disallow: \/');\n            zeilen.push('');\n        });\n\n        zeilen.push('Sitemap: ' + host + '\/sitemap.xml');\n\n        ergebnis.robots = zeilen.join('\\n');\n\n        \/\/ llms.txt ist Markdown und beschreibt, was auf der Seite steht.\n        const llms = [];\n        llms.push('# ' + name);\n        llms.push('');\n        llms.push('> ' + (String(e.zusammenfassung || '').trim() || T.llms_platzhalter));\n        llms.push('');\n        llms.push('## ' + T.llms_wichtig);\n        llms.push('');\n        llms.push('- [' + T.llms_start + '](' + host + '\/): ' + T.llms_start_text);\n        llms.push('- [' + T.llms_blog + '](' + host + '\/blog\/): ' + T.llms_blog_text);\n        llms.push('- [' + T.llms_ueber + '](' + host + '\/about\/): ' + T.llms_ueber_text);\n        llms.push('');\n        llms.push('## ' + T.llms_optional);\n        llms.push('');\n        llms.push('- [' + T.llms_sitemap + '](' + host + '\/sitemap.xml)');\n        ergebnis.llms = llms.join('\\n');<\/p>\n<p>        \/\/ --- Befunde ---\n        if (!erlaubt.suche) {\n            ergebnis.befunde.push({ schwer: true, text: T.b_suche_gesperrt });\n        }\n        if (!erlaubt.training && erlaubt.assistent) {\n            ergebnis.befunde.push({ schwer: false, text: T.b_training_ohne_assistent });\n        }\n        if (!erlaubt.training) {\n            ergebnis.befunde.push({ schwer: false, text: T.b_google_extended });\n            ergebnis.befunde.push({ schwer: false, text: T.b_rueckwirkend });\n        }\n        if (!erlaubt.assistent) {\n            ergebnis.befunde.push({ schwer: true, text: T.b_assistent_gesperrt });\n        }\n        if (pfade.some(function (p) { return p === '\/'; })) {\n            ergebnis.befunde.push({ schwer: true, text: T.b_alles_gesperrt });\n        }\n        if (pfade.some(function (p) { return p.indexOf('*') !== -1 || p.indexOf('$') !== -1; })) {\n            ergebnis.befunde.push({ schwer: false, text: T.b_platzhalter });\n        }\n        ergebnis.befunde.push({ schwer: false, text: T.b_kein_schutz });\n        ergebnis.befunde.push({ schwer: false, text: T.b_freiwillig });<\/p>\n<p>        ergebnis.gesperrt = gesperrt;\n        ergebnis.ok = true;\n        return ergebnis;\n    }<\/p>\n<p>    if (typeof window !== 'undefined') {\n        window.LW_TEST = window.LW_TEST || {};\n        window.LW_TEST.RB = { bauen: bauen, pfadeLesen: pfadeLesen, BOTS: BOTS };\n    }<\/p>\n<p>    \/\/ --- Anbindung an das Dokument ---<\/p>\n<p>    const btn = document.getElementById('rb-btn');\n    const out = document.getElementById('rb-ergebnis');\n    const gruppenFeld = document.getElementById('rb-gruppen');\n    if (!btn || !out || !gruppenFeld) { return; }<\/p>\n<p>    const KASTEN = 'background: var(--bg-body, #14141a); border: 1px solid var(--border, #2a2a35); border-radius: 6px; padding: 14px; margin-bottom: 18px;';\n    const UEBERSCHRIFT = 'color: var(--text-primary, #e8e8ee); font-weight: 700; margin-bottom: 8px;';<\/p>\n<p>    function el(tag, css, text) {\n        const knoten = document.createElement(tag);\n        if (css) { knoten.setAttribute('style', css); }\n        if (text !== undefined && text !== null) { knoten.textContent = text; }\n        return knoten;\n    }<\/p>\n<p>    GRUPPEN.forEach(function (gruppe) {\n        const zeile = el('label', 'display: flex; gap: 10px; align-items: flex-start; padding: 7px 0; cursor: pointer;');\n        const kasten = document.createElement('input');\n        kasten.type = 'checkbox';\n        kasten.id = 'rb-' + gruppe;\n        kasten.checked = gruppe === 'suche';\n        kasten.setAttribute('style', 'margin-top: 3px;');\n        zeile.appendChild(kasten);\n        const text = el('span', '');\n        text.appendChild(el('div', 'color: var(--text-primary, #e8e8ee); font-size: 0.9rem;', T['gruppe_' + gruppe]));\n        text.appendChild(el('div', 'color: var(--text-secondary, #a0a0b0); font-size: 0.8rem; line-height: 1.4;', T['gruppe_' + gruppe + '_text']));\n        zeile.appendChild(text);\n        gruppenFeld.appendChild(zeile);\n    });<\/p>\n<p>    function codeKasten(titel, inhalt, dateiname) {\n        const teil = document.createDocumentFragment();\n        teil.appendChild(el('div', UEBERSCHRIFT, titel));\n        teil.appendChild(el('div', 'color: var(--text-secondary, #a0a0b0); font-size: 0.82rem; margin-bottom: 8px;', dateiname));\n        teil.appendChild(el('pre', 'background: var(--bg-body, #14141a); border: 1px solid var(--border, #2a2a35); color: var(--text-primary, #e8e8ee); padding: 16px; border-radius: 6px; overflow-x: auto; font-family: monospace; font-size: 0.82rem; line-height: 1.55; white-space: pre; margin-bottom: 18px;', inhalt));\n        return teil;\n    }<\/p>\n<p>    btn.addEventListener('click', function () {\n        const eingabe = {\n            host: (document.getElementById('rb-host') || {}).value,\n            name: (document.getElementById('rb-name') || {}).value,\n            zusammenfassung: (document.getElementById('rb-zusammenfassung') || {}).value,\n            pfade: (document.getElementById('rb-pfade') || {}).value\n        };\n        GRUPPEN.forEach(function (g) {\n            const knoten = document.getElementById('rb-' + g);\n            eingabe[g] = knoten ? knoten.checked : false;\n        });<\/p>\n<p>        const erg = bauen(eingabe);\n        out.textContent = '';\n        if (!erg.ok) {\n            out.appendChild(el('div', 'background: var(--bg-body, #14141a); border: 1px solid var(--border, #2a2a35); border-left: 3px solid #ffb86c; color: var(--text-secondary, #a0a0b0); padding: 14px; border-radius: 6px;', erg.fehler));\n            return;\n        }<\/p>\n<p>        out.appendChild(codeKasten(T.h_robots, erg.robots, '\/robots.txt'));\n        out.appendChild(codeKasten(T.h_llms, erg.llms, '\/llms.txt'));<\/p>\n<p>        out.appendChild(el('div', UEBERSCHRIFT, T.h_kennungen));\n        const tabelle = el('div', KASTEN);\n        GRUPPEN.forEach(function (gruppe) {\n            const bots = erg.bots.filter(function (b) { return b.gruppe === gruppe; });\n            if (!bots.length) { return; }\n            tabelle.appendChild(el('div', 'color: var(--text-primary, #e8e8ee); font-size: 0.85rem; font-weight: 700; padding: 8px 0 4px;', T['gruppe_' + gruppe]));\n            const reihe = el('div', 'display: flex; gap: 8px; flex-wrap: wrap; padding-bottom: 6px;');\n            bots.forEach(function (b) {\n                reihe.appendChild(el('span', 'font-family: monospace; font-size: 0.78rem; padding: 3px 8px; border-radius: 4px; border: 1px solid var(--border, #2a2a35); color: ' + (b.erlaubt ? 'var(--accent, #7ee787)' : '#ffb86c') + ';', b.name));\n            });\n            tabelle.appendChild(reihe);\n        });\n        out.appendChild(tabelle);<\/p>\n<p>        out.appendChild(el('div', UEBERSCHRIFT, T.h_befunde));\n        const befunde = el('div', KASTEN);\n        erg.befunde.forEach(function (b) {\n            const z = el('div', 'display: flex; gap: 10px; padding: 5px 0;');\n            z.appendChild(el('span', 'color: ' + (b.schwer ? '#ffb86c' : 'var(--text-secondary, #a0a0b0)') + '; font-family: monospace;', b.schwer ? '!' : '-'));\n            z.appendChild(el('span', 'color: var(--text-secondary, #a0a0b0); font-size: 0.9rem; line-height: 1.5;', b.text));\n            befunde.appendChild(z);\n        });\n        out.appendChild(befunde);\n    });\n})();\n<\/script><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Writes a robots.txt that separates search engines, AI training crawlers and AI assistants, plus an llms.txt skeleton, and explains what each block really achieves.<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":38,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"template-tool-base.php","meta":{"footnotes":""},"tags":[91106,91386,91115],"class_list":["post-10253","page","type-page","status-publish","hentry","tag-automation","tag-marketing-cyfrowy-pl","tag-web-security"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>robots.txt &amp; llms.txt Generator for AI Crawlers - Lukas Wojcik - Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-llms-txt-ai-crawler-generator\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"robots.txt &amp; llms.txt Generator for AI Crawlers - Lukas Wojcik - Blog\" \/>\n<meta property=\"og:description\" content=\"Writes a robots.txt that separates search engines, AI training crawlers and AI assistants, plus an llms.txt skeleton, and explains what each block really achieves.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-llms-txt-ai-crawler-generator\/\" \/>\n<meta property=\"og:site_name\" content=\"Lukas Wojcik - Blog\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.lukaswojcik.com\/blog\/wp-content\/uploads\/2026\/08\/og-default.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/en\\\/toolbox\\\/robots-llms-txt-ai-crawler-generator\\\/\",\"url\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/en\\\/toolbox\\\/robots-llms-txt-ai-crawler-generator\\\/\",\"name\":\"robots.txt & llms.txt Generator for AI Crawlers - Lukas Wojcik - Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/#website\"},\"datePublished\":\"2026-08-14T22:42:16+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/en\\\/toolbox\\\/robots-llms-txt-ai-crawler-generator\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/en\\\/toolbox\\\/robots-llms-txt-ai-crawler-generator\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/en\\\/toolbox\\\/robots-llms-txt-ai-crawler-generator\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Toolbox\",\"item\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/en\\\/toolbox\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"robots.txt &#038; llms.txt Generator for AI Crawlers\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/\",\"name\":\"Lukas Wojcik - Blog\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/#\\\/schema\\\/person\\\/895f7604f9b6b71aad9bba33af28d0f9\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/#\\\/schema\\\/person\\\/895f7604f9b6b71aad9bba33af28d0f9\",\"name\":\"luky\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/lw-x2.jpg\",\"url\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/lw-x2.jpg\",\"contentUrl\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/lw-x2.jpg\",\"width\":424,\"height\":636,\"caption\":\"luky\"},\"logo\":{\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/lw-x2.jpg\"},\"sameAs\":[\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"robots.txt & llms.txt Generator for AI Crawlers - Lukas Wojcik - Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-llms-txt-ai-crawler-generator\/","og_locale":"en_US","og_type":"article","og_title":"robots.txt & llms.txt Generator for AI Crawlers - Lukas Wojcik - Blog","og_description":"Writes a robots.txt that separates search engines, AI training crawlers and AI assistants, plus an llms.txt skeleton, and explains what each block really achieves.","og_url":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-llms-txt-ai-crawler-generator\/","og_site_name":"Lukas Wojcik - Blog","og_image":[{"width":1200,"height":630,"url":"https:\/\/www.lukaswojcik.com\/blog\/wp-content\/uploads\/2026\/08\/og-default.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-llms-txt-ai-crawler-generator\/","url":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-llms-txt-ai-crawler-generator\/","name":"robots.txt & llms.txt Generator for AI Crawlers - Lukas Wojcik - Blog","isPartOf":{"@id":"https:\/\/www.lukaswojcik.com\/blog\/#website"},"datePublished":"2026-08-14T22:42:16+00:00","breadcrumb":{"@id":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-llms-txt-ai-crawler-generator\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-llms-txt-ai-crawler-generator\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-llms-txt-ai-crawler-generator\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.lukaswojcik.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Toolbox","item":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/"},{"@type":"ListItem","position":3,"name":"robots.txt &#038; llms.txt Generator for AI Crawlers"}]},{"@type":"WebSite","@id":"https:\/\/www.lukaswojcik.com\/blog\/#website","url":"https:\/\/www.lukaswojcik.com\/blog\/","name":"Lukas Wojcik - Blog","description":"","publisher":{"@id":"https:\/\/www.lukaswojcik.com\/blog\/#\/schema\/person\/895f7604f9b6b71aad9bba33af28d0f9"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.lukaswojcik.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/www.lukaswojcik.com\/blog\/#\/schema\/person\/895f7604f9b6b71aad9bba33af28d0f9","name":"luky","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.lukaswojcik.com\/blog\/wp-content\/uploads\/2026\/07\/lw-x2.jpg","url":"https:\/\/www.lukaswojcik.com\/blog\/wp-content\/uploads\/2026\/07\/lw-x2.jpg","contentUrl":"https:\/\/www.lukaswojcik.com\/blog\/wp-content\/uploads\/2026\/07\/lw-x2.jpg","width":424,"height":636,"caption":"luky"},"logo":{"@id":"https:\/\/www.lukaswojcik.com\/blog\/wp-content\/uploads\/2026\/07\/lw-x2.jpg"},"sameAs":["https:\/\/www.lukaswojcik.com\/blog"]}]}},"_links":{"self":[{"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/pages\/10253","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/comments?post=10253"}],"version-history":[{"count":0,"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/pages\/10253\/revisions"}],"up":[{"embeddable":true,"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/pages\/38"}],"wp:attachment":[{"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/media?parent=10253"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/tags?post=10253"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}