{"id":15605,"date":"2026-10-30T08:05:00","date_gmt":"2026-10-30T07:05:00","guid":{"rendered":"https:\/\/www.lukaswojcik.com\/blog\/?p=15605"},"modified":"2026-10-26T13:13:14","modified_gmt":"2026-10-26T12:13:14","slug":"verifying-crawler-addresses-reverse-dns-published-ranges-and-where-each-fails","status":"publish","type":"post","link":"https:\/\/www.lukaswojcik.com\/blog\/en\/digital-marketing\/tutorials-en-digital-marketing\/verifying-crawler-addresses-reverse-dns-published-ranges-and-where-each-fails\/","title":{"rendered":"Verifying Crawler Addresses: Reverse DNS, Published Ranges, and Where Each Fails"},"content":{"rendered":"

The User-Agent is a client-supplied label naming the browser or automated program. A user agent such as Googlebot<\/code> starts as a claim supplied by the client. The requested path also comes from the request. The connection address determined by the server is stronger evidence, provided any upstream proxy has been handled correctly. Only then can the address be compared with the named service’s documented infrastructure.<\/p>\n

Two usual checks are a documented DNS naming pattern with forward confirmation and a match against official IP ranges. A registry entry such as \u201cGoogle LLC\u201d or \u201cMicrosoft Corporation\u201d adds context but does not identify a particular crawler.<\/p>\n

\"Crawler
A range match confirms the published infrastructure category. Historical examples are not a current allowlist.<\/figcaption><\/figure>\n

DNS verification: name, boundary and forward check<\/h2>\n

A reverse lookup returns a PTR name for an IP address. That name alone is insufficient: the address block’s administrator can set a PTR without controlling the domain named in the record. Verification therefore needs three steps:<\/p>\n

  1. The PTR lookup starts with the actual source address.<\/li>
  2. The name is compared with the currently documented pattern for the appropriate crawler category. Google’s common crawlers, special crawlers and user-triggered fetchers are distinct categories.<\/li>
  3. The accepted name is resolved through A or AAAA records. The response must contain the original address.<\/li><\/ol>\n

    The domain boundary matters: crawl.example.googlebot.com<\/code> is beneath googlebot.com<\/code>; googlebot.com.example.org<\/code> is not. An arbitrary successful DNS round trip establishes an address-to-name relationship, not Googlebot identity. A google.com<\/code> suffix alone is also insufficient for that specific classification.<\/p>\n

    This method does not require a downloaded range list, but still needs maintenance. Naming patterns can change, DNS responses expire, and lookup failures must remain distinct from negative responses.<\/p>\n

    Range lists and the correct category<\/h2>\n

    Official JSON files contain IPv4 and, for some services, IPv6 prefixes in CIDR notation, such as 66.249.66.0\/27<\/code>. Verification tests whether an address falls within a network. A plain text comparison is insufficient.<\/p>\n

    Google now links to common-crawlers.json<\/code> for common crawlers, a separate special-crawler list, and several lists for user-triggered fetchers and agents. A match first establishes membership of the published infrastructure category. A shared list does not necessarily distinguish every individual user agent.<\/p>\n

    OpenAI publishes separate lists for GPTBot, OAI-SearchBot and ChatGPT-User. The purposes differ: model training, search and certain user actions. A missing PTR is therefore not automatically suspicious. For services without a documented DNS verification method, the corresponding official list is the relevant evidence.<\/p>\n

    What the historical sample establishes<\/h2>\n

    The saved dataset from 5 September 2026<\/strong> contains twelve queried lists with 2,560 prefix entries; 2,143 belonged to four Google lists available at the time. These totals were recalculated from the saved counts during this revision. Prefix entries are neither unique IP addresses nor necessarily disjoint networks. The count does not reliably establish traffic volume, architecture or update frequency.<\/p>\n

    For 20.171.207.1<\/code> and 52.230.152.1<\/code>, the historical output records GPTBot range matches without a resolved PTR. The saved field does not distinguish an absent PTR from a lookup failure. The Microsoft registry entry also does not prove a particular cloud tenancy agreement. All address findings remain historical observations, not a current allowlist.<\/p>\n

    8.8.8.8<\/code> illustrates the difference between organisation and function: Google documents this address as a public DNS resolver. A successful DNS round trip and Google registry entry do not make it Googlebot. \u201cGooglebot\u201d was the selected test claim in that check, not evidence of an actual HTTP request from the resolver.<\/p>\n

    64.233.173.1<\/code> appeared in the historical Google user-triggered fetcher list and had a corresponding google-proxy<\/code> name. That establishes neither a specific visitor action nor automatic Googlebot identity. The tool’s classification was too broad at precisely this point.<\/p>\n

    Identity and access policy are separate decisions<\/h2>\n

    robots.txt<\/code> controls cooperative crawlers and remains useful for that purpose. It is not an access barrier. A verified operator does not mean every service follows the same robots.txt rules: Google distinguishes common crawlers, special cases and user-triggered fetchers. Googlebot does not support crawl-delay<\/code>, so that directive does not distinguish Googlebot from a user-triggered fetch.<\/p>\n

    Missing or contradictory verification does not automatically establish malicious behaviour. Depending on the purpose, unwanted requests can be addressed through authentication, rate limits or rules at the web server or upstream protection layer. An IP block is one option, not a complete defence. There is no universal shared crawl budget covering every operator.<\/p>\n

    A reproducible manual cross-check<\/h2>\n

    The example uses an IPv4 address shown in Google’s documentation. The PTR name must come from the actual response; category and name-boundary checks precede forward resolution. Downloading the JSON file alone does not perform a CIDR match.<\/p>\n

    Shell<\/span>