{"id":20964,"date":"2026-10-01T18:26:42","date_gmt":"2026-10-01T16:26:42","guid":{"rendered":"https:\/\/www.lukaswojcik.com\/blog\/?p=20964"},"modified":"2026-10-01T18:29:42","modified_gmt":"2026-10-01T16:29:42","slug":"sitemap-structures-24-sites-index-splitting-extensions-violations","status":"publish","type":"post","link":"https:\/\/www.lukaswojcik.com\/blog\/en\/web-development\/sitemap-structures-24-sites-index-splitting-extensions-violations\/","title":{"rendered":"Sitemap Structures Measured at 24 Sites: Indexes, Splitting, Extensions and Violations"},"content":{"rendered":"

On a small site a sitemap is a list. On a large one it is a building made of files: an index that points to child files, a split by content type, language or running number, and extensions for images, videos, news and language versions. Each of these decisions has a limit in the protocol, and each can go wrong without any page visibly breaking.<\/p>\n

For this article the sitemaps of 24 domains were read on 1 October 2026 and checked with the core of the new Sitemap Checker & Validator<\/a>: search, infrastructure, news, public administration, retail and travel, in three countries, plus this site.<\/p>\n

\n\"Five
Structures, violations and limits in the sitemaps of 18 readable domains, measured on 1 October 2026.<\/figcaption><\/figure>\n

How the measurement was done<\/h2>\n

The sitemap was located the way a crawler finds it: first the first Sitemap:<\/code> line in robots.txt<\/code>, otherwise \/sitemap.xml<\/code> and \/sitemap_index.xml<\/code>. Where the file was an index, three child files were fetched: the first, one from the middle and the last. Compressed files (.gz<\/code>) were decompressed. The check was run by the same code that runs in the tool in the browser, here under Node on this site’s own server.<\/p>\n

Whether the addresses in the sitemaps are reachable was not checked. That is a different question with its own traps, and it is what the XML Sitemap Auditor<\/a> is for, which fetches a sample of the addresses. This article is about the files themselves: structure, format and the rules of sitemaps.org and Google.<\/p>\n

Six domains without a readable sitemap<\/h2>\n

Of 24 domains, 18 delivered a readable sitemap. Four name no sitemap in robots.txt<\/code> and answer 404 at \/sitemap.xml<\/code>, one of them only after a redirect. Two turned the fetching client away with 403, one of them on the very file its own robots.txt<\/code> names.<\/p>\n

One case lies in between. heise.de names seven sitemaps in robots.txt<\/code>, and the first of them, \/sitemapindex.xml<\/code>, answers 404. The standard address \/sitemap.xml<\/code>, on the other hand, delivers a file with 10,000 entries. A crawler that reads every line loses nothing; a tool that takes only the first reports a missing sitemap.<\/p>\n

robots.txt as a table of contents<\/h2>\n

The protocol allows any number of Sitemap:<\/code> lines, and that is exactly how several large sites work: instead of an index, the list of files sits directly in robots.txt<\/code>.<\/p>\n\n\n\n\n\n\n\n\n\n\n
Domain<\/th>\nSitemap:<\/code> lines<\/th>\n<\/tr>\n
booking.com<\/td>\n434<\/td>\n<\/tr>\n
nytimes.com<\/td>\n25<\/td>\n<\/tr>\n
microsoft.com<\/td>\n22<\/td>\n<\/tr>\n
otto.de<\/td>\n19<\/td>\n<\/tr>\n
bbc.co.uk<\/td>\n13<\/td>\n<\/tr>\n
wordpress.org<\/td>\n9<\/td>\n<\/tr>\n
heise.de<\/td>\n7<\/td>\n<\/tr>\n
seven more<\/td>\n1 to 5<\/td>\n<\/tr>\n<\/table>\n

This has an advantage that no specification spells out: every line is also proof that the operator of the host wants this file. Under sitemaps.org, a sitemap may contain addresses of another host only if that host’s robots.txt<\/code> points to it.<\/p>\n

Five ways to build a sitemap<\/h2>\n

The 18 sitemaps fall into five structures, and some sites mix two of them.<\/p>\n\n\n\n\n\n\n\n
Structure<\/th>\nExamples from the measurement<\/th>\n<\/tr>\n
one file<\/td>\nheise.de (10,000 entries), lukaswojcik.com (1,006), Cloudflare (933); the news sitemaps of the New York Times, Spiegel and the Guardian, each named first in its robots.txt<\/code><\/td>\n<\/tr>\n
index by content type<\/td>\nwordpress.org (pages, images, videos), Otto (49 categories), Airbnb (homes, experiences, destinations, help pages)<\/td>\n<\/tr>\n
index by language or market<\/td>\nIkea with 2,166 files following the pattern prod-et-EE_1.xml<\/code>, MDN with ten languages, Booking with 45 languages per content type, Shopify with 568 files<\/td>\n<\/tr>\n
index by running number<\/td>\ngov.uk with 35 files of up to 25,000 addresses each, Airbnb with 140 files for homes alone<\/td>\n<\/tr>\n
index of indexes<\/td>\nApple (47 country indexes, one of them on apple.com.cn), Microsoft (five sub-indexes), one child file at Google<\/td>\n<\/tr>\n<\/table>\n

The last structure deserves a second look. sitemaps.org describes an index as a list of sitemaps, and Google’s guide to large sitemaps says nothing about nesting. An index that points to further indexes therefore relies on behaviour nobody promises. Apple’s root index also names files on a second host, which under the protocol is covered only by that host’s robots.txt<\/code>.<\/p>\n

The limit reached first<\/h2>\n

A sitemap may contain at most 50,000 addresses and may be at most 50 MB uncompressed. gov.uk stays at half with 25,000 addresses per file; one Airbnb file sits exactly on the limit with 50,000. At Ikea the other limit applies.<\/p>\n

The file prod-et-EE_1.xml<\/code> contains only 4,617 addresses but weighs 47.6 MB, 95.3 percent of what is allowed. Each address carries on average 69 xhtml:link<\/code> entries for its language versions, 319,635 in total, plus 22,797 images and 1,585 videos. Around 10.8 KB per entry is no longer the address but its description.<\/p>\n

This is the arithmetic hreflang in a sitemap brings with it: every version of a page lists all other versions and itself. Across all files the number of links therefore grows with the square of the language versions, not with the number of pages. For sitemaps with hreflang the byte limit is the better yardstick for splitting than the number of addresses.<\/p>\n

Where the violations sit<\/h2>\n

In the core format of loc<\/code>, lastmod<\/code>, changefreq<\/code> and priority<\/code> the validator found little: a few duplicate entries, three addresses with a fragment (#<\/code>) and two on another host at heise.de, and empty child files at Shopify and Ikea. Almost all violations of weight sit in the namespaces and the extensions.<\/p>\n