Sitemap Structures Measured at 24 Sites: Indexes, Splitting, Extensions and Violations

A sitemap lists a website’s URLs for search engines. On larger sites, an index distributes that list across several files. Extra fields describe images, videos, news or language versions. Errors in these files may be invisible on the website itself but can make processing harder for search engines.

On 1 October 2026, 24 domains were examined with the core of the Sitemap Checker & Validator. The selection included news, public administration, retail, travel and this website. The figures below describe that historical sample, not a current complete audit of every site.

How the measurement was done

The measurement script followed a simple discovery strategy: first the first Sitemap: entry in robots.txt. Only without such an entry did the script try /sitemap.xml and, after a failed request, /sitemap_index.xml. For an index, at most three child files were selected: first, middle and last. This is not a crawler’s complete discovery process. Gzip content was decompressed, and the checking core ran under Node on the site’s server.

The measurement checked sitemap files, not the reachability of every listed page. The XML Sitemap Auditor checks a limited selection of destination URLs for that purpose. These are different questions: is the list structured correctly, and do selected destinations return useful responses?

Six domains without a readable sitemap

The surviving log initially contains 17 readable starting files, six 404 responses and one 403. A separately saved Heise file brings the usable sample to 18 domains. That leaves five 404 responses and one 403 refusal among the other six. This only means that the chosen discovery process found no readable file at the time; a sitemap elsewhere remains possible.

At heise.de, the first of seven robots.txt entries pointed to /sitemapindex.xml, which returned 404. A separate request saved a file with 10,000 entries from /sitemap.xml. Other robots.txt entries covered different sections. Reading every entry broadens discovery but guarantees neither identical coverage nor a successful request for each file.

robots.txt as a table of contents

Multiple Sitemap: lines in robots.txt are allowed. Large sites can announce several lists directly without putting a shared index in front. The table counts entries in the saved measurement log.

A robots.txt entry can also confirm a sitemap hosted elsewhere. The sitemaps.org protocol explicitly describes that method. Google additionally permits combined submission for sites whose ownership has been verified in Search Console. Without knowing that configuration, a different host alone therefore does not prove an error.

Five ways to build a sitemap

The usable files show five recurring ways of splitting sitemaps. The examples describe only the files examined; a domain may use other structures at the same time.

An index containing other indexes is explicitly an error for Google: Search Console help requires direct references to sitemap files. This is stronger than the absence of a promise in the main guide. Apple’s reference to another host raises a separate question: a cross-site submission setup was outside this measurement and cannot be assumed or ruled out from the files alone.

The limit reached first

Both limits apply at once: no more than 50,000 URLs and no more than 52,428,800 uncompressed bytes. The protocol calls that byte limit “50 MB”; technically it is 50 MiB. The examined gov.uk files contained up to 25,000 URLs, while one Airbnb file contained exactly 50,000.

The saved Ikea file prod-et-EE_1.xml contains 4,617 URLs and 49,954,185 bytes: about 47.64 MiB and 95.28 percent of the limit. It also contains 319,635 language references, 22,797 images and 1,585 videos. That averages about 69 language references and 10.8 kB of total file size per URL. The latter includes the entire XML share, not just descriptions.

hreflang links language versions of the same page. With three versions, each version lists all three destinations: nine references in total. Ten versions produce a hundred. For M content groups with L versions each, the count is M × L². Size therefore grows with the number of content groups as well as the square of the version count. Splitting must consider both URL count and actual bytes.

Rule violations and other findings

The validator reported different kinds of findings. Duplicate URLs, fragments and empty files need a different assessment from missing required fields or a wrong XML namespace. The saved Heise file, for example, contains three fragment URLs and 16 repeated URLs. The following list separates specific rule violations from findings that merely need investigation.

• Google root index: The log reports the old namespace http://www.google.com/schemas/sitemap/0.84 instead of http://www.sitemaps.org/schemas/sitemap/0.9. A strict checker designed for version 0.9 does not recognise this as the expected document.
• Gmail sitemap: Among 9,075 language references for 166 URLs, 165 used es-419. Google explicitly lists that regional code as unsupported. This is a limit of Google’s hreflang processing, not a claim that the language tag is invalid in general.
• Cloudflare: The sitemap and XHTML namespaces begin with https:// instead of the specified http://. These are different namespace identifiers. The 10,263 affected language references therefore do not match the expected XHTML namespace; this does not establish how every possible parser handles them.
• wordpress.org: The measurement log reports missing video:description for all 62 videos, although Google requires that field. In the image file, 521 of 693 entries repeat a URL already listed. This is avoidable duplication: one URL entry can contain up to 1,000 images. Duplication alone does not make the XML invalid.
• New York Times: For 13 of 745 news entries, news:language is en-US. Google requires a two- or three-letter language code here, so English uses en; the documented special cases are zh-cn and zh-tw.
• BBC: The index lists three files about the 2014 elections. One examined file contains 204 HTTP URLs. Neither the topic’s age nor HTTP alone is a sitemap-format error. Whether those destinations now redirect or are obsolete was not checked in this file audit.

These findings do not all belong to the same error category. A wrong namespace or missing required field violates a format rule; an old HTTP link or repeated URL alone does not. The files also cannot show which pages Google actually indexes or omits because of a particular finding.

lastmod: one date for everything

Of the three optional core fields lastmod, priority and changefreq, Google documents using only lastmod, when the values are reliable. Parts of the sample still contain the other two fields. A field’s presence therefore does not establish a benefit for Google.

Identical modification dates invite investigation but do not prove incorrect data. Cloudflare had one date across 933 entries, the large Ikea file across 4,617, and the examined Booking language files across 3,032 each. A generator run could explain this, but so could a genuine common change. Many different values do not establish accuracy either. Checking accuracy requires comparison with actual page changes. Omitting lastmod is allowed.

In the historical comparison on this site, the Yoast index gave 28 September for post-sitemap.xml, while an entry inside the child file carried 1 October. That suggests inconsistent timestamps. The index date should describe a change to the sitemap file; a URL entry describes the individual page. A reliable comparison needs both files and, ideally, the actual modification times.

The location rule and the XSD

The location rule also depends on submission. Without special submission, a sitemap at /sitemaps/sitemap.xml applies to paths below that directory. Several sampled files also listed URLs outside that scope. Google permits a wider scope when the sitemap is submitted through Search Console. That configuration was not checked, so these findings require clarification rather than proving that almost every large site breaks the rules.

An XSD is a machine-readable description of the permitted XML structure. The basic sitemaps.org schema uses processContents="strict" to require known declarations for foreign extensions. Without an additional image schema, even an otherwise correct image:image therefore fails. An isolated libxml2 2.9.14 test confirmed this. Another test showed differing timestamp rules: the XSD rejects 2026-10-01T14:05 but accepts 2026-10-01T14:05:00. The linked W3C date profile requires a timezone when a time is present. A date alone, or a timestamp with seconds and timezone, avoids that difference.

Checking a sitemap before it is published

The Sitemap Checker & Validator examines pasted XML or a file in the browser. Gzip files require the browser’s decompression feature used by the tool. Selected structural and content rules are checked; a green result is not a complete audit against every search-engine rule. The checks include:

• XML syntax errors such as an unescaped & or a truncated file, with a reported location,
• expected namespaces, encoding declarations, URL count and file size,
• core fields such as loc and lastmod and the ordering specified by the XSD,
• hreflang codes and relationships between URLs contained in the same file,
• selected required fields and limits for images, videos and news,
• the location rule when the sitemap URL is supplied.

The local file checker does not upload the supplied XML for validation. The XML Sitemap Auditor, in contrast, fetches a limited selection of published URLs. Its HTML noindex and canonical detection is incomplete: recognition partly depends on attribute order, and X-Robots-Tag is not checked. An unremarkable result therefore guarantees neither indexability nor correct annotations on every page.

Context and limits

The selection covers 24 domains on one day and at most three child files per starting index. At Ikea, that examines only three of 2,166 files. This is neither a random sample nor representative of all sitemaps. After the additional Heise request, six domains remained without a usable file. A 403 for the test client also does not prove that Googlebot receives the same status. Search-result effects were not measured.