English
Deutsch
PolskiSave tool
A robots.txt file is not read top to bottom. Each crawler first picks exactly one group, and inside that group the longest matching pattern wins, with Allow beating Disallow on a tie. This tool applies those three rules from RFC 9309 and names the winning line for every combination.
Matching ignores case, and a name in the file counts when the identifier starts with it.
Paths only, starting with a slash. A query string may be included.
The address is fetched once from this server. Only the calling IP address, the time and the size of the request are stored, for seven days; the address entered is not.
Eight queries per hour. Only /robots.txt is fetched, at most 500 KiB, private address ranges are refused.