LW IT Solutions

Well-known & Policy Files Inspector: security.txt, ads.txt, gpc.json, llms.txt

A domain speaks to machines through a handful of files at fixed paths. security.txt tells researchers where to report a vulnerability, robots.txt and llms.txt address crawlers, ads.txt names who may sell the inventory, gpc.json declares that Global Privacy Control is honoured, change-password sends password managers to the right page, and assetlinks and the Apple app association bind apps to the site. This check fetches all thirteen from this server and judges each by its own specification: a file that exists but has expired, is served as HTML, or answers 200 with the home page instead of a 404, is worth knowing about.

Thirteen requests of at most 200 KB each go from this server to the host entered, following up to three redirects each. Hosts in private, loopback and link-local networks are refused. The query is protected by reCAPTCHA v3; the calling IP address and the target are stored for one hour to limit the rate.

Limits worth knowing: only the fixed paths are checked, not files a site keeps elsewhere; files are cut at 200 KB; a soft 404 is recognised by HTML in the body, which misses servers that answer with a JSON error page; the security.txt signature is detected but not verified; and the absence of most of these files is normal for most sites, so absent counts as information, not as a fault.

OTHER TOOLS