Bot Detection: IP Lists and Cryptographic Signatures Compared
Apple enlarged the address pool behind Applebot during August 2026. The reported figures are a jump from around 2,400 addresses across twelve prefixes to roughly 7,056 across thirty-three, and the published range list is described as still incomplete. Every site that verifies that crawler against a stored copy of the list had to fetch a new one, or quietly started treating a legitimate crawler as an impostor.
That is not a flaw in this particular release. It is the defining property of proving identity by address, and it is worth setting against the alternative rather than complaining about it.
The address proof, and its self-updating variant
The simplest form is a list of ranges the operator publishes and every site downloads. It works because a source address cannot be forged across a completed TCP handshake, which makes it a genuine proof of origin rather than a claim.
Forward-confirmed reverse DNS is the same proof with the maintenance moved. Instead of holding the ranges, a site resolves the address backwards, then resolves the resulting name forwards again and checks that it lands on the address it started from.
address in the request 17.241.75.12
reverse lookup -> some-host.applebot.apple.com
suffix check ends in a domain the operator owns
forward lookup -> 17.241.75.12
compare same address, so the claim holds
fails safe: a forged reverse record does not survive the
forward lookup, because the attacker does not control the
operator's forward zoneThis variant costs one or two DNS round trips per new address, cached afterwards, and it does not go stale when the pool grows. It remains an address proof, though, with everything that implies.
What an expansion actually costs
A pool that nearly triples produces two failure modes, and only one of them is loud. The loud one is a crawler being blocked, noticed within a day because content stops being indexed. The quiet one is worse: a site that treats unverified traffic as ordinary human traffic now has several thousand addresses’ worth of crawler activity sitting in its analytics as sessions.
Neither failure announces itself as a list problem. Both look like something else, which is why the refresh cadence for any published range list belongs in a scheduled job rather than in someone’s memory.
The key proof
The other approach moves the evidence into the request. The operator holds a key pair, publishes the public half under a domain it controls, and signs each outgoing request; the site verifies the signature against the published key. HTTP message signatures, standardised in RFC 9421, are the mechanism, and Web Bot Auth is the profile of it aimed at agents.
Its advantage over an address is structural. A key follows the operator, not the machine, so new capacity signs with the same key on the day it comes online and no list anywhere needs touching. It also survives rented infrastructure, where an address genuinely cannot help: the address block belongs to a hosting provider, and knowing that tells nobody who is renting it this month.
The cost is that it only exists where an operator has chosen to implement it. An address proof works against every crawler that publishes ranges, which is most of the established ones; a key proof works against the subset that signs, which today is small.
Which one, and when
In practice they are not competitors. An address check answers a question a signature cannot: whether traffic claiming to be a well-known crawler really originates where that crawler lives. A signature answers a question an address cannot: which of several operators sharing a data centre actually sent this.
The sensible arrangement runs the address check first, because it is cheap and covers the established crawlers, and accepts a valid signature as an independent second path for operators that offer one. What neither of them provides is a judgement about what the request is for. Both establish who is asking. Nothing in either proof speaks to what happens with the answer.
Related articles
