Bot Detection: IP Lists and Cryptographic Signatures Compared

An HTTP request claiming to come from Applebot or Googlebot is not authenticated by its User-Agent name. Two useful checks are an address matched against the crawler provider’s published information and a valid HTTP message signature tied to a trusted agent identity. Both need maintenance, and neither automatically grants access.

Apple publishes an Applebot CIDR list and documents DNS verification. A CIDR prefix describes an address range. A stored list can miss new ranges after an expansion. The source date and the most recent successful refresh therefore belong in the verification log.

What an IP check establishes

A match associates the observed source address with a range published for that crawler. A hosting company’s entire network is not an equivalent allowlist. Conversely, rented infrastructure can support address verification when the crawler provider publishes sufficiently specific ranges.

A completed TCP connection makes simple off-path source-address spoofing difficult. This is not an unconditional identity guarantee: a proxy changes the visible peer, infrastructure can be compromised, and HTTP/3 uses QUIC rather than TCP. The useful evidence is the network endpoint combined with reliable provider information.

DNS verification without a locally stored range list

Forward-confirmed reverse DNS combines a reverse lookup, a provider-specific hostname check and a forward lookup. A reverse record alone is insufficient. The following sequence is illustrative; no live lookup of the sample address is asserted.

observed client IP: 192.0.2.10 (documentation address)
1. PTR lookup produces a hostname.
2. Normalized hostname has the exact suffix .applebot.apple.com
   with a nonempty host label before the suffix.
3. A/AAAA lookup includes the original client IP.
4. Only then: address verification succeeds.

Rejected: fakeapplebot.apple.com
Rejected: host.applebot.apple.com.attacker.example
Timeout or mismatch: verification not established.

The accepted domain must come from that crawler’s documentation, not from any domain mentioned in the request. Case normalization, a final DNS dot and exact label boundaries matter. DNS caching needs bounded expiry; timeouts and negative results need an explicit policy. DNS verification avoids a local CIDR snapshot but still depends on correct, available DNS. The number of network lookups varies with caches and DNS resolution.

Maintenance and failure handling

Range downloads need an authoritative HTTPS source, validation of the returned CIDRs, an atomic update and monitoring of age and failures. Replacing a valid list with an empty download can block legitimate traffic. Retaining the previous list after a failure also has a cost: newly added or removed ranges remain unrecognized until refresh. The acceptable stale period belongs in the access policy.

A stale list can cause a blocked crawler or an incorrect analytics classification. Neither outcome is guaranteed, and a specific indexing impact need not become visible within a day. A failed verification means “unverified”, not “human” or necessarily “malicious”.

What a signature adds

RFC 9421 defines signatures over selected HTTP message components. Web Bot Auth adds discovery and binding of verification keys for automated traffic. As of 24 October 2026, the relevant IETF protocol remains an Internet-Draft, not a finished RFC. Implementation profiles can differ; Google’s experimental integration and Cloudflare’s documented integration cannot simply be assumed interchangeable.

A successful check binds the covered content to a key associated with the accepted agent identifier. A self-published key does not by itself prove a company’s legal identity or permission to read protected content. Trust in the identifier and access rules remain separate decisions. A stolen key also invalidates the assumption that only the intended agent can sign.

Signatures can remain useful across changes of hosting or source IP. However, key rotation, revocation, cache expiry, safe key retrieval and compatible verification still require work at both ends. Google explicitly notes that not every request from participating agents is signed. Missing signatures therefore cannot alone identify an impostor.

Choosing the verification path

There is no universal requirement to check IP before signatures. The applicable provider, available evidence, latency and access policy determine the order. Results are best recorded separately: address verified, signature verified, verification failed or unavailable, and access allowed or denied. Authentication is one input to a decision; rate limits and content restrictions still apply.

Practical check: analysis of GA4 traffic groups

The GA4 traffic review tool groups reported data by browser, operating system, country and session source/medium. It shows explicit cues instead of a bot probability: no engaged sessions, zero average duration, missing dimensions or an automation-related browser name. A configurable minimum keeps small groups from receiving these cues, while retaining their rows in the output.

These observations do not authenticate a crawler. A campaign, consent behaviour, incomplete collection or internal testing can produce similar patterns. Country and Direct traffic are not evidence of automation. The exported CSV can support a comparison with server or CDN logs, but this report contains neither client IP addresses nor verified signatures. No data is excluded or changed.

Google automatically excludes known bot traffic and does not disclose its excluded count in GA4. A result without matching rules therefore means only that the selected checks found nothing in the returned rows. Sources: Google’s known-bot exclusion and Data API schema.