{"id":15718,"date":"2026-09-06T10:53:58","date_gmt":"2026-09-06T08:53:58","guid":{"rendered":"https:\/\/www.lukaswojcik.com\/blog\/toolbox\/robots-txt-crawler-matrix\/"},"modified":"2026-09-06T10:53:58","modified_gmt":"2026-09-06T08:53:58","slug":"robots-txt-crawler-matrix","status":"publish","type":"page","link":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-txt-crawler-matrix\/","title":{"rendered":"robots.txt Crawler Matrix: which rule wins for which bot"},"content":{"rendered":"<div class=\"gtm-analyser-container\" style=\"background: var(--bg-panel, #1e1e24); padding: 25px; border-radius: 8px; border: 1px solid var(--border, #2a2a35);\">\n<p style=\"color: var(--text-secondary, #a0a0b0); margin-bottom: 20px;\">A robots.txt file is not read top to bottom. Each crawler first picks exactly one group, and inside that group the longest matching pattern wins, with Allow beating Disallow on a tie. This tool applies those three rules from RFC 9309 and names the winning line for every combination.<\/p>\n<div style=\"margin-bottom: 14px;\">\n        <label for=\"rb-url\" style=\"color: var(--text-secondary, #a0a0b0); display: block; font-size: 0.85rem; margin-bottom: 5px;\">Domain or address<\/label><br \/>\n        <input id=\"rb-url\" type=\"text\" value=\"nytimes.com\" class=\"form-control\" style=\"width: 100%; padding: 10px; background: var(--bg-body, #14141a); border: 1px solid var(--border, #2a2a35); color: var(--text-primary, #e8e8ee); border-radius: 6px; box-sizing: border-box; font-family: monospace;\">\n    <\/div>\n<div style=\"display: grid; grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); gap: 14px; margin-bottom: 14px;\">\n<div>\n            <label for=\"rb-agenten\" style=\"color: var(--text-secondary, #a0a0b0); display: block; font-size: 0.85rem; margin-bottom: 5px;\">Crawler identifiers, one per line (up to 8)<\/label><br \/>\n            <textarea id=\"rb-agenten\" rows=\"8\" class=\"form-control\" style=\"width: 100%; padding: 10px; background: var(--bg-body, #14141a); border: 1px solid var(--border, #2a2a35); color: var(--text-primary, #e8e8ee); border-radius: 6px; box-sizing: border-box; font-family: monospace; font-size: 0.85rem;\">Googlebot<br \/>\nBingbot<br \/>\nGPTBot<br \/>\nClaudeBot<br \/>\nGoogle-Extended<br \/>\nCCBot<br \/>\nPerplexityBot<br \/>\nApplebot-Extended<\/textarea><\/p>\n<p style=\"color: var(--text-secondary, #a0a0b0); font-size: 0.78rem; margin: 5px 0 0;\">Matching ignores case, and a name in the file counts when the identifier starts with it.<\/p>\n<\/p><\/div>\n<div>\n            <label for=\"rb-pfade\" style=\"color: var(--text-secondary, #a0a0b0); display: block; font-size: 0.85rem; margin-bottom: 5px;\">Paths to test, one per line (up to 12)<\/label><br \/>\n            <textarea id=\"rb-pfade\" rows=\"8\" class=\"form-control\" style=\"width: 100%; padding: 10px; background: var(--bg-body, #14141a); border: 1px solid var(--border, #2a2a35); color: var(--text-primary, #e8e8ee); border-radius: 6px; box-sizing: border-box; font-family: monospace; font-size: 0.85rem;\">\/<br \/>\n\/wp-admin\/<br \/>\n\/wp-login.php<br \/>\n\/wp-json\/<br \/>\n\/?s=test<br \/>\n\/feed\/<br \/>\n\/cgi-bin\/<br \/>\n\/admin\/<\/textarea><\/p>\n<p style=\"color: var(--text-secondary, #a0a0b0); font-size: 0.78rem; margin: 5px 0 0;\">Paths only, starting with a slash. A query string may be included.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<p style=\"color: var(--text-secondary, #a0a0b0); font-size: 0.82rem; margin-bottom: 16px;\">The address is fetched once from this server. Only that request is stored, for the hourly limit.<\/p>\n<p>    <button id=\"rb-btn\" class=\"button\" style=\"background: var(--accent, #7ee787); color: var(--on-accent, #0b1114); border: none; padding: 12px 24px; border-radius: 6px; font-weight: 700; cursor: pointer;\">Read robots.txt<\/button><\/p>\n<div id=\"rb-ausgabe\" style=\"margin-top: 22px;\"><\/div>\n<p style=\"color: var(--text-secondary, #a0a0b0); font-size: 0.8rem; margin: 24px 0 0;\">Eight queries per hour. Only \/robots.txt is fetched, at most 500 KiB, private address ranges are refused.<\/p>\n<\/div>\n<p><script>\n(function () {\n    'use strict';<\/p>\n<p>    const T = {\"einleitung\":\"A robots.txt file is not read top to bottom. Each crawler first picks exactly one group, and inside that group the longest matching pattern wins, with Allow beating Disallow on a tie. This tool applies those three rules from RFC 9309 and names the winning line for every combination.\",\"label_url\":\"Domain or address\",\"label_agenten\":\"Crawler identifiers, one per line (up to 8)\",\"label_pfade\":\"Paths to test, one per line (up to 12)\",\"hinweis_agenten\":\"Matching ignores case, and a name in the file counts when the identifier starts with it.\",\"hinweis_pfade\":\"Paths only, starting with a slash. A query string may be included.\",\"datenschutz\":\"The address is fetched once from this server. Only that request is stored, for the hourly limit.\",\"knopf\":\"Read robots.txt\",\"grenze\":\"Eight queries per hour. Only \\\/robots.txt is fetched, at most 500 KiB, private address ranges are refused.\",\"laeuft\":\"Fetching and evaluating the rules file \u2026\",\"urteil_gut\":\"Nothing stands in the way\",\"urteil_warnung\":\"Worth a look\",\"urteil_kritisch\":\"Needs attention\",\"wirkung_regeln_gelten\":\"The file was delivered and its rules apply.\",\"wirkung_alles_erlaubt\":\"No rules file is available. A 4xx answer counts as absent, and an absent file means every path is open.\",\"wirkung_nichts_crawlen\":\"The server answered with an error. A 5xx answer means unreachable, and a crawler then treats the whole site as disallowed for a while.\",\"wirkung_unerreichbar\":\"The file could not be fetched at all.\",\"wirkung_unklar\":\"The answer fits none of the cases RFC 9309 describes.\",\"h_abruf\":\"The request\",\"h_matrix\":\"Matrix\",\"h_regeln\":\"Which rule decided\",\"h_befunde\":\"Findings\",\"zeile_datei\":\"Address\",\"zeile_status\":\"Status\",\"zeile_spruenge\":\"Redirects\",\"zeile_typ\":\"Content type\",\"zeile_groesse\":\"Size\",\"zeile_gruppen\":\"Groups in the file\",\"zeile_sitemaps\":\"Sitemap declarations\",\"zeile_dauer\":\"Duration\",\"zeile_limit\":\"Queries left this hour\",\"spalte_kennung\":\"Crawler\",\"spalte_gruppe\":\"Group\",\"spalte_pfad\":\"Path\",\"spalte_regel\":\"Rule\",\"spalte_zeile\":\"Line\",\"legende\":\"A plus sign means crawling is allowed, a minus sign means it is not.\",\"via_stern\":\"* (fallback)\",\"keine_gruppe\":\"none\",\"keine_befunde\":\"No findings.\",\"stufe_hoch\":\"High\",\"stufe_mittel\":\"Medium\",\"stufe_niedrig\":\"Low\",\"stufe_info\":\"Note\",\"stufe_gut\":\"Good\",\"befund_nicht_erreichbar\":\"The rules file could not be fetched || The request failed with: {fehler}. Whether a crawler treats this as absent or as unreachable depends on the exact cause, and the two lead to opposite behaviour.\",\"befund_status_410\":\"410 Gone on robots.txt || A 410 is a 4xx answer, and RFC 9309 counts every 4xx as no rules file at all. The effect is the opposite of what the status code suggests: instead of closing the site, it opens every path.\",\"befund_status_gesperrt\":\"Status {status} on robots.txt || Access to the rules file itself is refused. As a 4xx answer this counts as absent, so all paths are treated as allowed.\",\"befund_status_4xx\":\"Status {status} on robots.txt || There is no rules file. Every path counts as allowed.\",\"befund_status_5xx\":\"Status {status} on robots.txt || A server error means unreachable, not absent. Crawlers then hold off on the whole site, and Google keeps doing so for around twelve hours before falling back to a cached copy.\",\"befund_status_unklar\":\"Unusual status {status} || Neither the success nor the error case of RFC 9309 applies here.\",\"befund_weitergeleitet\":\"Redirected {spruenge} times || The rules file was reached at {endurl}. RFC 9309 has crawlers follow at most five redirects; beyond that the file counts as absent.\",\"befund_typ_nicht_text\":\"Content type is {typ} || The rules file should be delivered as text\\\/plain. Some crawlers refuse anything else.\",\"befund_zu_gross\":\"File is {bytes} bytes || Google reads at most {grenze} bytes and ignores everything after that.\",\"befund_leer\":\"The file is empty || An empty rules file allows everything, which is the same result as no file at all.\",\"befund_bom\":\"Byte order mark at the start || Crawlers skip it, but it is a sign the file was written by an editor that adds one.\",\"befund_keine_sterngruppe\":\"No group for * || Every crawler without a group of its own has no rules to follow at all.\",\"befund_gruppe_doppelt\":\"{agent} appears in more than one group || Crawlers merge such groups differently, so which rules apply is not defined. Groups: {gruppen}.\",\"befund_kennung_wiederholt\":\"{agent} is listed {anzahl} times in the same group || This changes nothing about the result, it is only repetition.\",\"befund_muster_ohne_schraegstrich\":\"Line {zeile}: pattern without a leading slash || The value {muster} starts with neither \\\/ nor *, so the rule matches nothing.\",\"befund_zeile_ohne_doppelpunkt\":\"Line {zeile} has no colon || The line reads {text} and is ignored.\",\"befund_zeile_regel_ohne_gruppe\":\"Line {zeile} stands before any group || A rule needs a user-agent line above it, otherwise it belongs to nothing.\",\"befund_crawl_delay\":\"Crawl-delay is set {anzahl} times || Google ignores this field entirely; Bing and Yandex honour it.\",\"befund_felder_ohne_wirkung\":\"Fields without effect: {felder} || These are not part of RFC 9309, and most crawlers skip them.\",\"befund_sitemap_fehlt\":\"No sitemap declaration || A Sitemap line is the only way a crawler finds the index without guessing at addresses.\",\"befund_sitemap_relativ\":\"Sitemap without a full address || The value {wert} is not an absolute URL, and the line has no effect.\",\"befund_sitemap_fremder_host\":\"Sitemap on another host: {hosts} || This is allowed, but the addresses in it only count for a host whose ownership the crawler also recognises.\",\"befund_sitemap_vorhanden\":\"{anzahl} sitemap declarations\",\"befund_kennung_ganz_gesperrt\":\"Fully blocked for every path tested: {agenten} || This is a decision, not a defect, and it is listed here so the decision stays visible.\",\"fehler_adresse\":\"That address cannot be used.\",\"fehler_netz\":\"The server could not be reached.\",\"fehler_antwort\":\"The answer was not readable.\",\"fehler_captcha\":\"The spam check failed. A reload usually helps.\",\"fehler_kein_token\":\"The spam check did not produce a token.\",\"fehler_zu_viele\":\"Hourly limit of {limit} queries reached.\",\"fehler_zaehler\":\"The counter is unavailable.\",\"fehler_pruefdienst\":\"The spam check is unavailable.\",\"fehler_aufbau\":\"The tool is misconfigured.\",\"fehler_methode\":\"Wrong request method.\",\"fehler_eingabe\":\"The input was not readable.\",\"fehler_unbekannt\":\"Something went wrong.\",\"grund_leer\":\"No address was given.\",\"grund_zu_lang\":\"The address is too long.\",\"grund_schema\":\"Only http and https work.\",\"grund_unlesbar\":\"The address could not be parsed.\",\"grund_benutzerinfo\":\"Addresses with a user name are refused.\",\"grund_gesperrter_bereich\":\"That address points into a private network.\",\"grund_kein_punkt\":\"A host name needs a dot.\",\"grund_hostform\":\"A domain name is required, not an IP address.\",\"grund_port\":\"That port is not allowed.\",\"grund_kein_dns\":\"The name does not resolve.\"};<\/p>\n<p>    var ENDPUNKT = '\/lw-robots.php';\n    var SITEKEY = '6LcrPkEtAAAAAPo1QCOf-IIM2fCL0UfdJz4y2iSY';\n    var STUFEN = ['hoch', 'mittel', 'niedrig', 'info', 'gut'];\n    var WIRKUNGEN = ['regeln_gelten', 'alles_erlaubt', 'nichts_crawlen', 'unerreichbar', 'unklar'];\n    var GRUENDE = ['regel', 'keine_regel', 'keine_gruppe', 'status'];<\/p>\n<p>    function liste(w) { return Array.isArray(w) ? w : []; }\n    function zahl(w) { return (typeof w === 'number' && isFinite(w)) ? w : null; }\n    function text(schluessel, daten) {\n        var t = T[schluessel];\n        if (typeof t !== 'string') { return null; }\n        return t.replace(\/\\{([a-z_0-9]+)\\}\/g, function (m, k) {\n            return (daten && daten[k] !== undefined && daten[k] !== null) ? String(daten[k]) : m;\n        });\n    }\n    function kb(bytes) { if (bytes === null) { return '-'; } if (bytes < 1024) { return bytes + ' B'; } return (Math.round(bytes \/ 102.4) \/ 10) + ' KB'; }\n\n    \/** Zeilen eines Textfeldes zu einer Liste machen. *\/\n    function zeilen(roh, grenze) {\n        return String(roh || '').split(\/\\r?\\n\/).map(function (z) { return z.trim(); })\n            .filter(function (z) { return z !== ''; }).slice(0, grenze);\n    }\n\n    function bewerten(antwort) {\n        var a = antwort || {};\n        if (a.fehler) { return { fehler: String(a.fehler), grund: a.grund || null, limit: zahl(a.limit) }; }\n        var befunde = liste(a.befunde).map(function (f) {\n            return { key: String(f.key || ''), stufe: STUFEN.indexOf(f.stufe) === -1 ? 'info' : f.stufe, daten: (f.daten &#038;&#038; typeof f.daten === 'object') ? f.daten : {} };\n        });\n        var gruppen = {};\n        STUFEN.forEach(function (s) { gruppen[s] = befunde.filter(function (f) { return f.stufe === s; }); });\n        var abruf = (a.abruf &#038;&#038; typeof a.abruf === 'object') ? a.abruf : {};\n        var matrix = liste(a.matrix).map(function (z) {\n            z = z || {};\n            return {\n                agent: String(z.agent || ''),\n                wodurch: z.wodurch ? String(z.wodurch) : null,\n                stern: !!z.stern,\n                felder: liste(z.felder).map(function (f) {\n                    f = f || {};\n                    var r = (f.regel &#038;&#038; typeof f.regel === 'object') ? f.regel : null;\n                    return {\n                        pfad: String(f.pfad || ''),\n                        erlaubt: !!f.erlaubt,\n                        grund: GRUENDE.indexOf(f.grund) === -1 ? 'status' : f.grund,\n                        regel: r ? { art: r.art === 'allow' ? 'allow' : 'disallow', muster: String(r.muster || ''), zeile: zahl(r.zeile) } : null\n                    };\n                })\n            };\n        });\n        return {\n            fehler: null,\n            basis: a.basis ? String(a.basis) : '',\n            url: a.url ? String(a.url) : '',\n            urteil: (a.urteil === 'gut' || a.urteil === 'warnung' || a.urteil === 'kritisch') ? a.urteil : 'kritisch',\n            wirkung: WIRKUNGEN.indexOf(a.wirkung) === -1 ? 'unklar' : a.wirkung,\n            abruf: {\n                status: zahl(abruf.status), fehler: abruf.fehler ? String(abruf.fehler) : null,\n                spruenge: zahl(abruf.spruenge) || 0, endurl: abruf.endurl ? String(abruf.endurl) : null,\n                typ: abruf.typ ? String(abruf.typ) : null, bytes: zahl(abruf.bytes)\n            },\n            gruppenZahl: zahl(a.gruppen_gesamt) || 0,\n            sitemaps: liste(a.sitemaps).length,\n            pfade: liste(a.pfade).map(String),\n            matrix: matrix,\n            befunde: befunde, stufengruppen: gruppen,\n            dauer: zahl(a.dauer_ms), limitRest: zahl(a.limit_rest)\n        };\n    }\n\n    function skriptLaden() {\n        return new Promise(function (auf) {\n            if (window.grecaptcha) { auf(true); return; }\n            var s = document.createElement('script');\n            s.src = 'https:\/\/www.google.com\/recaptcha\/api.js?render=' + SITEKEY;\n            s.onload = function () { auf(true); }; s.onerror = function () { auf(false); };\n            document.head.appendChild(s);\n            setTimeout(function () { auf(!!window.grecaptcha); }, 8000);\n        });\n    }\n    function tokenHolen() {\n        return skriptLaden().then(function (da) {\n            if (!da || !window.grecaptcha || !window.grecaptcha.ready) { return null; }\n            return new Promise(function (auf) {\n                var fertig = false;\n                function einmal(w) { if (!fertig) { fertig = true; auf(w); } }\n                setTimeout(function () { einmal(null); }, 12000);\n                try {\n                    window.grecaptcha.ready(function () {\n                        if (fertig) { return; }\n                        if (!window.grecaptcha.execute) { einmal(null); return; }\n                        window.grecaptcha.execute(SITEKEY, { action: 'robots' }).then(function (t) { einmal(t || null); }, function () { einmal(null); });\n                    });\n                } catch (e) { einmal(null); }\n            });\n        }, function () { return null; });\n    }\n    function abfragen(url, agenten, pfade) {\n        return tokenHolen().then(function (token) {\n            return fetch(ENDPUNKT, { method: 'POST', headers: { 'Content-Type': 'application\/json' },\n                body: JSON.stringify({ url: url, agenten: agenten, pfade: pfade, token: token || '' }) })\n                .then(function (r) { return r.json().then(function (j) { return j; }, function () { return { fehler: 'antwort' }; }); }, function () { return { fehler: 'netz' }; });\n        });\n    }\n\n    window.LW_TEST = window.LW_TEST || {};\n    window.LW_TEST.RB = { bewerten: bewerten, darstellen: null, text: text, kb: kb, zeilen: zeilen,\n        ENDPUNKT: ENDPUNKT, STUFEN: STUFEN, WIRKUNGEN: WIRKUNGEN, GRUENDE: GRUENDE, T: T };\n\n    var btn = document.getElementById('rb-btn');\n    var out = document.getElementById('rb-ausgabe');\n    if (!btn || !out) { return; }\n\n    function el(tag, stil, txt) { var e = document.createElement(tag); if (stil) { e.setAttribute('style', stil); } if (txt !== undefined &#038;&#038; txt !== null) { e.textContent = txt; } return e; }\n    function wert(id) { var e = document.getElementById(id); return e ? e.value : ''; }\n    var UEBERSCHRIFT = 'font-family: \"Nunito Sans\", sans-serif; font-weight: 700; color: var(--text-primary, #e8e8ee); font-size: 0.95rem; margin: 22px 0 8px;';\n    var ZELLE = 'padding: 5px 12px 5px 0; color: var(--text-secondary, #a0a0b0); font-size: 0.87rem;';\n    var ZELLE_WERT = 'padding: 5px 12px 5px 0; color: var(--text-primary, #e8e8ee); font-size: 0.87rem; font-family: monospace; word-break: break-all;';\n    var KASTEN = 'background: var(--bg-body, #14141a); border: 1px solid var(--border, #2a2a35); border-radius: 8px; padding: 14px 16px; margin: 0 0 10px;';\n    var GUT = ' color: #7ee787;', WARN = ' color: #ffa94d;', ROT = ' color: #ff7b72;', GRAU = ' color: #8a8a99;';\n    var FARBEN = { hoch: '#ff7b72', mittel: '#ffa94d', niedrig: '#e3b341', info: '#8a8a99', gut: '#7ee787' };\n    var URTEIL = { gut: GUT, warnung: WARN, kritisch: ROT };\n    function gitter(sp) { return el('div', 'display: grid; grid-template-columns: ' + sp + '; gap: 0 18px; align-items: baseline;'); }\n    function paar(g, n, w, stil) { g.appendChild(el('div', ZELLE, n)); g.appendChild(el('div', ZELLE_WERT + (stil || ''), w)); }\n\n    function darstellen(b, ziel) {\n        ziel.innerHTML = '';\n        if (b.fehler) {\n            var txt = T['fehler_' + b.fehler] || T.fehler_unbekannt;\n            if (b.grund &#038;&#038; T['grund_' + b.grund]) { txt = txt + ' ' + T['grund_' + b.grund]; }\n            if (b.limit !== null) { txt = txt.replace('{limit}', String(b.limit)); }\n            ziel.appendChild(el('p', ZELLE + WARN + ' margin: 0;', txt));\n            return;\n        }\n\n        ziel.appendChild(el('div', 'font-size: 1.05rem; font-weight: 700; margin: 0 0 6px;' + URTEIL[b.urteil], T['urteil_' + b.urteil]));\n        ziel.appendChild(el('p', ZELLE + ' margin: 0 0 12px;', T['wirkung_' + b.wirkung]));\n\n        \/\/ --- Der Abruf selbst\n        ziel.appendChild(el('div', UEBERSCHRIFT, T.h_abruf));\n        var ga = gitter('auto auto');\n        paar(ga, T.zeile_datei, b.url, GRAU);\n        paar(ga, T.zeile_status, b.abruf.status === null ? (b.abruf.fehler || '-') : String(b.abruf.status),\n            b.abruf.status !== null &#038;&#038; b.abruf.status >= 200 && b.abruf.status < 300 ? GUT : WARN);\n        if (b.abruf.spruenge > 0) { paar(ga, T.zeile_spruenge, String(b.abruf.spruenge) + (b.abruf.endurl ? ' \u2192 ' + b.abruf.endurl : ''), GRAU); }\n        if (b.abruf.typ) { paar(ga, T.zeile_typ, b.abruf.typ, GRAU); }\n        if (b.abruf.bytes !== null && b.abruf.bytes > 0) { paar(ga, T.zeile_groesse, kb(b.abruf.bytes), GRAU); }\n        paar(ga, T.zeile_gruppen, String(b.gruppenZahl), GRAU);\n        paar(ga, T.zeile_sitemaps, String(b.sitemaps), GRAU);\n        if (b.dauer !== null) { paar(ga, T.zeile_dauer, b.dauer + ' ms', GRAU); }\n        if (b.limitRest !== null) { paar(ga, T.zeile_limit, String(b.limitRest), GRAU); }\n        ziel.appendChild(ga);<\/p>\n<p>        \/\/ --- Die Matrix\n        ziel.appendChild(el('div', UEBERSCHRIFT, T.h_matrix));\n        var huelle = el('div', 'overflow-x: auto;');\n        var spalten = 'auto auto' + b.pfade.map(function () { return ' auto'; }).join('');\n        var gm = gitter(spalten);\n        var kopf = [T.spalte_kennung, T.spalte_gruppe].concat(b.pfade);\n        kopf.forEach(function (s) {\n            gm.appendChild(el('div', ZELLE + ' font-weight: 700; color: var(--text-primary, #e8e8ee); white-space: nowrap; font-family: monospace;', s));\n        });\n        b.matrix.forEach(function (z) {\n            gm.appendChild(el('div', ZELLE_WERT + ' white-space: nowrap;', z.agent));\n            gm.appendChild(el('div', ZELLE_WERT + GRAU + ' white-space: nowrap;',\n                z.wodurch === null ? T.keine_gruppe : (z.stern ? T.via_stern : z.wodurch)));\n            z.felder.forEach(function (f) {\n                gm.appendChild(el('div', ZELLE_WERT + (f.erlaubt ? GUT : ROT) + ' text-align: center;', f.erlaubt ? '+' : '\u2212'));\n            });\n        });\n        huelle.appendChild(gm);\n        ziel.appendChild(huelle);\n        ziel.appendChild(el('p', ZELLE + GRAU + ' margin: 6px 0 0;', T.legende));<\/p>\n<p>        \/\/ --- Welche Regel entschieden hat\n        var entschieden = [];\n        b.matrix.forEach(function (z) {\n            z.felder.forEach(function (f) {\n                if (f.grund === 'regel' && f.regel) { entschieden.push({ agent: z.agent, pfad: f.pfad, regel: f.regel, erlaubt: f.erlaubt }); }\n            });\n        });\n        if (entschieden.length) {\n            ziel.appendChild(el('div', UEBERSCHRIFT, T.h_regeln));\n            var h2 = el('div', 'overflow-x: auto;');\n            var gr = gitter('auto auto auto auto');\n            [T.spalte_kennung, T.spalte_pfad, T.spalte_regel, T.spalte_zeile].forEach(function (s) {\n                gr.appendChild(el('div', ZELLE + ' font-weight: 700; color: var(--text-primary, #e8e8ee); white-space: nowrap;', s));\n            });\n            entschieden.slice(0, 80).forEach(function (e) {\n                gr.appendChild(el('div', ZELLE_WERT + ' white-space: nowrap;', e.agent));\n                gr.appendChild(el('div', ZELLE_WERT + ' white-space: nowrap;', e.pfad));\n                gr.appendChild(el('div', ZELLE_WERT + (e.erlaubt ? GUT : ROT) + ' white-space: nowrap;', e.regel.art + ': ' + (e.regel.muster === '' ? '(leer)' : e.regel.muster)));\n                gr.appendChild(el('div', ZELLE_WERT + GRAU, e.regel.zeile === null ? '-' : String(e.regel.zeile)));\n            });\n            h2.appendChild(gr);\n            ziel.appendChild(h2);\n        }<\/p>\n<p>        \/\/ --- Befunde\n        ziel.appendChild(el('div', UEBERSCHRIFT, T.h_befunde));\n        var irgendwas = false;\n        STUFEN.forEach(function (s) {\n            var gr2 = b.stufengruppen[s];\n            if (!gr2.length) { return; }\n            irgendwas = true;\n            ziel.appendChild(el('div', 'font-weight: 700; font-size: 0.82rem; text-transform: uppercase; letter-spacing: 0.5px; margin: 14px 0 6px; color: ' + FARBEN[s] + ';', T['stufe_' + s] + ' (' + gr2.length + ')'));\n            gr2.forEach(function (f) {\n                var k = el('div', KASTEN + ' border-left: 3px solid ' + FARBEN[s] + ';');\n                var d = {};\n                Object.keys(f.daten).forEach(function (n) {\n                    var w = f.daten[n];\n                    d[n] = Array.isArray(w) ? w.join(', ') : w;\n                });\n                var t = text('befund_' + f.key, d) || f.key;\n                var teile = t.split(' || ');\n                k.appendChild(el('div', 'color: var(--text-primary, #e8e8ee); font-size: 0.9rem; font-weight: 700;', teile[0]));\n                if (teile[1]) { k.appendChild(el('div', ZELLE + ' padding: 6px 0 0; line-height: 1.55;', teile[1])); }\n                ziel.appendChild(k);\n            });\n        });\n        if (!irgendwas) { ziel.appendChild(el('p', ZELLE + ' margin: 0;', T.keine_befunde)); }\n    }\n    window.LW_TEST.RB.darstellen = darstellen;<\/p>\n<p>    var laeuft = false;\n    btn.addEventListener('click', function () {\n        if (laeuft) { return; }\n        laeuft = true; btn.disabled = true;\n        out.innerHTML = '';\n        out.appendChild(el('p', ZELLE + GRAU + ' margin: 0;', T.laeuft));\n        abfragen(wert('rb-url'), zeilen(wert('rb-agenten'), 8), zeilen(wert('rb-pfade'), 12)).then(\n            function (a) { darstellen(bewerten(a), out); laeuft = false; btn.disabled = false; },\n            function () { darstellen(bewerten({ fehler: 'netz' }), out); laeuft = false; btn.disabled = false; });\n    });\n})();\n<\/script><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Reads a site&#8217;s robots.txt, resolves every user agent to the group that governs it, and shows for each path whether crawling is allowed and which rule decided it. Includes the part most summaries get wrong: what a 404, a 410 or a 500 on the rules file itself means for a crawler.<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":38,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"template-tool-base.php","meta":{"footnotes":""},"tags":[91380,91094,92757],"class_list":["post-15718","page","type-page","status-publish","hentry","tag-artificial-intelligence","tag-seo","tag-web-development"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>robots.txt Crawler Matrix: which rule wins for which bot - Lukas Wojcik - Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-txt-crawler-matrix\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"robots.txt Crawler Matrix: which rule wins for which bot - Lukas Wojcik - Blog\" \/>\n<meta property=\"og:description\" content=\"Reads a site&#039;s robots.txt, resolves every user agent to the group that governs it, and shows for each path whether crawling is allowed and which rule decided it. Includes the part most summaries get wrong: what a 404, a 410 or a 500 on the rules file itself means for a crawler.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-txt-crawler-matrix\/\" \/>\n<meta property=\"og:site_name\" content=\"Lukas Wojcik - Blog\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.lukaswojcik.com\/blog\/wp-content\/uploads\/2026\/08\/og-default.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/en\\\/toolbox\\\/robots-txt-crawler-matrix\\\/\",\"url\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/en\\\/toolbox\\\/robots-txt-crawler-matrix\\\/\",\"name\":\"robots.txt Crawler Matrix: which rule wins for which bot - Lukas Wojcik - Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/#website\"},\"datePublished\":\"2026-09-06T08:53:58+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/en\\\/toolbox\\\/robots-txt-crawler-matrix\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/en\\\/toolbox\\\/robots-txt-crawler-matrix\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/en\\\/toolbox\\\/robots-txt-crawler-matrix\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Toolbox\",\"item\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/en\\\/toolbox\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"robots.txt Crawler Matrix: which rule wins for which bot\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/\",\"name\":\"Lukas Wojcik - Blog\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/#\\\/schema\\\/person\\\/895f7604f9b6b71aad9bba33af28d0f9\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/#\\\/schema\\\/person\\\/895f7604f9b6b71aad9bba33af28d0f9\",\"name\":\"luky\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/lw-x2.jpg\",\"url\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/lw-x2.jpg\",\"contentUrl\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/lw-x2.jpg\",\"width\":424,\"height\":636,\"caption\":\"luky\"},\"logo\":{\"@id\":\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/lw-x2.jpg\"},\"sameAs\":[\"https:\\\/\\\/www.lukaswojcik.com\\\/blog\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"robots.txt Crawler Matrix: which rule wins for which bot - Lukas Wojcik - Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-txt-crawler-matrix\/","og_locale":"en_US","og_type":"article","og_title":"robots.txt Crawler Matrix: which rule wins for which bot - Lukas Wojcik - Blog","og_description":"Reads a site's robots.txt, resolves every user agent to the group that governs it, and shows for each path whether crawling is allowed and which rule decided it. Includes the part most summaries get wrong: what a 404, a 410 or a 500 on the rules file itself means for a crawler.","og_url":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-txt-crawler-matrix\/","og_site_name":"Lukas Wojcik - Blog","og_image":[{"width":1200,"height":630,"url":"https:\/\/www.lukaswojcik.com\/blog\/wp-content\/uploads\/2026\/08\/og-default.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-txt-crawler-matrix\/","url":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-txt-crawler-matrix\/","name":"robots.txt Crawler Matrix: which rule wins for which bot - Lukas Wojcik - Blog","isPartOf":{"@id":"https:\/\/www.lukaswojcik.com\/blog\/#website"},"datePublished":"2026-09-06T08:53:58+00:00","breadcrumb":{"@id":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-txt-crawler-matrix\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-txt-crawler-matrix\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/robots-txt-crawler-matrix\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.lukaswojcik.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Toolbox","item":"https:\/\/www.lukaswojcik.com\/blog\/en\/toolbox\/"},{"@type":"ListItem","position":3,"name":"robots.txt Crawler Matrix: which rule wins for which bot"}]},{"@type":"WebSite","@id":"https:\/\/www.lukaswojcik.com\/blog\/#website","url":"https:\/\/www.lukaswojcik.com\/blog\/","name":"Lukas Wojcik - Blog","description":"","publisher":{"@id":"https:\/\/www.lukaswojcik.com\/blog\/#\/schema\/person\/895f7604f9b6b71aad9bba33af28d0f9"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.lukaswojcik.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/www.lukaswojcik.com\/blog\/#\/schema\/person\/895f7604f9b6b71aad9bba33af28d0f9","name":"luky","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.lukaswojcik.com\/blog\/wp-content\/uploads\/2026\/07\/lw-x2.jpg","url":"https:\/\/www.lukaswojcik.com\/blog\/wp-content\/uploads\/2026\/07\/lw-x2.jpg","contentUrl":"https:\/\/www.lukaswojcik.com\/blog\/wp-content\/uploads\/2026\/07\/lw-x2.jpg","width":424,"height":636,"caption":"luky"},"logo":{"@id":"https:\/\/www.lukaswojcik.com\/blog\/wp-content\/uploads\/2026\/07\/lw-x2.jpg"},"sameAs":["https:\/\/www.lukaswojcik.com\/blog"]}]}},"_links":{"self":[{"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/pages\/15718","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/comments?post=15718"}],"version-history":[{"count":0,"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/pages\/15718\/revisions"}],"up":[{"embeddable":true,"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/pages\/38"}],"wp:attachment":[{"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/media?parent=15718"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lukaswojcik.com\/blog\/wp-json\/wp\/v2\/tags?post=15718"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}