{"id":15617,"date":"2026-11-01T08:05:00","date_gmt":"2026-11-01T07:05:00","guid":{"rendered":"https:\/\/www.lukaswojcik.com\/blog\/?p=15617"},"modified":"2026-10-26T12:52:24","modified_gmt":"2026-10-26T11:52:24","slug":"security-txt-ads-txt-and-llms-txt-what-a-fixed-path-check-can-establish","status":"publish","type":"post","link":"https:\/\/www.lukaswojcik.com\/blog\/en\/web-development\/tutorials-en-web-development\/security-txt-ads-txt-and-llms-txt-what-a-fixed-path-check-can-establish\/","title":{"rendered":"security.txt, ads.txt and llms.txt: What a Fixed-Path Check Can Establish"},"content":{"rendered":"

Known paths make selected website information easier to find. A response at such a path does not by itself prove that the expected file is valid. The historical sample below records the output of a bounded checker, not a census of standards compliance.<\/p>\n

\"Historical
Historical classifications do not prove format validity.<\/figcaption><\/figure>\n

A registry, conventions and a proposal<\/h2>\n

RFC 8615 reserves \/.well-known\/ for application-defined resources. It does not make every resource public without conditions or require a blanket firewall exception. Each format has its own scope and retrieval rules.<\/p>\n

The sample also probes root paths. These do not all predate \/.well-known\/: ads.txt began in 2017 and llms.txt was proposed in September 2024. A sitemap need not be named sitemap.xml. The current llms.txt v2 proposal also allows a subpath such as \/docs\/llms.txt, so a root-only check cannot establish absence throughout a site.<\/p>\n

What the historical sample counted<\/h2>\n

The saved result is dated 5 September 2026, 18:19:59 +02:00. It contains twenty hosts and thirteen requested paths per host: 260 recorded path results. Redirects may require additional HTTP requests. The table reports the checker\u2019s classification as present; change-password uses its separate redirected classification.<\/p>\n

These counts do not establish that age caused adoption. The sample is selected rather than representative, and app-link or advertising files serve particular purposes. A missing file is not automatically a defect.<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Path<\/th>\nClassified results out of 20<\/th>\n<\/tr>\n<\/thead>\n
\/.well-known\/security.txt<\/code><\/td>\n14<\/td>\n<\/tr>\n
\/security.txt<\/code><\/td>\n5<\/td>\n<\/tr>\n
\/robots.txt<\/code><\/td>\n18<\/td>\n<\/tr>\n
\/ads.txt<\/code><\/td>\n9<\/td>\n<\/tr>\n
\/app-ads.txt<\/code><\/td>\n8<\/td>\n<\/tr>\n
\/.well-known\/gpc.json<\/code><\/td>\n2<\/td>\n<\/tr>\n
\/llms.txt<\/code><\/td>\n4<\/td>\n<\/tr>\n
\/.well-known\/change-password<\/code><\/td>\n4<\/td>\n<\/tr>\n
\/.well-known\/assetlinks.json<\/code><\/td>\n11<\/td>\n<\/tr>\n
\/.well-known\/apple-app-site-association<\/code><\/td>\n7<\/td>\n<\/tr>\n
\/.well-known\/traffic-advice<\/code><\/td>\n1<\/td>\n<\/tr>\n
\/humans.txt<\/code><\/td>\n4<\/td>\n<\/tr>\n
\/sitemap.xml<\/code><\/td>\n10<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Four overlaps, not five<\/h2>\n

Fourteen hosts were classified as having \/.well-known\/security.txt and five as having \/security.txt. Only four belonged to both groups: nytimes.com, bbc.co.uk, gov.uk and onet.pl. The fifth root result, heise.de, had a 404 at the well-known location in this capture.<\/p>\n

For web services, RFC 9116 places security.txt under \/.well-known\/. A root redirect can support older clients without maintaining two copies. A conforming file needs Contact and one Expires timestamp, HTTPS, text\/plain and UTF-8. The complete syntax and any Canonical field still need checking; the presence of two field names is not full validation.<\/p>\n

A redirect is not proof of a password-change form<\/h2>\n

The change-password convention directs clients to the account\u2019s password-change page. The snapshot has four redirected classifications, but two ended at status 301 after three recorded hops. The checker had not reached and verified a final form in those cases.<\/p>\n

Even a redirect ending in status 200 may lead to a login, homepage or unrelated page. A valid endpoint needs a suitable destination and flow; any redirect is not sufficient evidence.<\/p>\n

Refusal and HTML require interpretation<\/h2>\n

Stack Overflow returned one 418 and seven 429 responses, not proof of a single rule blocking every unknown client. These statuses do not establish whether the requested files existed. Other hosts also returned 403 or 406 responses.<\/p>\n

Two results were labelled soft 404 because HTML appeared where another format was expected. That is a rule of thumb based on the response text, not a confirmed diagnosis: challenge pages and legitimate explanatory HTML can resemble this pattern. Content-Type remains relevant to format compliance. Neither a status code nor the first part of a body alone settles every case.<\/p>\n

The inspector\u2019s limits<\/h2>\n

The linked inspector requests thirteen paths, allows up to three redirects and reads up to 204,800 bytes per resource. Its HTML heuristic examines only the first 2,000 characters and excludes the sitemap case. Local fixtures confirmed that plain error text and HTML at sitemap.xml can still be labelled present.<\/p>\n

Its parsers provide selected checks, not complete validation. A PGP marker is not signature verification. The link counter misses relative Markdown links, such as \/contact instead of a full https:\/\/ address. A redirect is not a tested password-change process. Truncated responses and these limits must remain part of the interpretation.<\/p>\n

GPC declarations and AI guidance<\/h2>\n

Two captures contain a positive GPC declaration, but one also records a missing update date. gpc.json is an optional statement of intended support; absence does not prove rejection, and presence does not prove actual processing behaviour. This small sample cannot measure general browser adoption.<\/p>\n

llms.txt offers structured guidance and links for agents. It is not an access-control mechanism or a guarantee that a crawler reads the file. ads.txt identifies authorised advertising sellers; a count of records does not independently verify each commercial relationship.<\/p>\n

Well-known and Policy Files Inspector<\/a><\/p>\n

\n

Questions and answers<\/h2>\n

Is HEAD enough to validate a file?<\/h3>\n

No. HEAD can inspect response headers but does not provide the body needed to check fields or syntax. A GET with recorded status, headers, redirect chain and body is more informative. A redirect-following command can issue several requests.<\/p>\n

Should every well-known path bypass security controls?<\/h3>\n

No. Access should match the specific resource and expected clients. A targeted exception for a public contact file is different from removing protection from the entire directory. A 403 or 429 result should remain distinct from a confirmed 404.<\/p>\n<\/div>\n

\n

Sources<\/h2>\n