LW IT Solutions logo LW IT Solutions logo LW IT Solutions

XML Sitemap Auditor: Index, lastmod and a Sample Check

Toolbox feedback

Bug, idea or suggestion

Feedback is stored on this server together with the tool page and language. No confidential data, please. Telegram receives only a notification without the feedback text.

Save tool

A sitemap is a promise to crawlers: these addresses exist, they are meant to be indexed, and this is when they last changed. Broken promises cost crawl budget and trust: listed pages that redirect or answer 404, noindex pages in the list, a lastmod that is the same for every entry, or a sitemap that no robots.txt announces. This check finds the sitemap the way a crawler does, reads an index and up to ten child sitemaps, judges the entries, and then fetches twelve of the listed pages spread across the list to compare promise and reality.

From this server: robots.txt, the sitemap and up to ten child sitemaps (5 MB each at most), then twelve listed pages of at most 64 KB each. Hosts in private, loopback and link-local networks are refused. The query is protected by reCAPTCHA v3; the calling IP address and the size of the request are stored for seven days, to limit the rate and to detect abuse; the input itself is not stored.

Limits worth knowing: only the first ten child sitemaps of an index are read, and only the first 20,000 entries are kept for the statistics; the sample is twelve pages spread evenly through the list, which finds systematic problems and misses single broken entries; noindex is read from the HTML only, not from an X-Robots-Tag header; and image, video and news extensions are counted as ordinary entries.

Articles that explain this

Other tools

All 145 tools

Related articles

LUKASWOJCIK.COM · NEWSLETTER

Fresh ideas in your inbox

New articles, practical tools and updates from the projects.

Choose topics (optional)
No selection: all topics.

Subscription starts after confirmation by email. · Privacy

Privacy · Newsletter

Email address, language, selected topics and subscription/confirmation dates are stored to deliver the newsletter. Double opt-in is required. Every issue includes an immediate unsubscribe link. Formconfirmed checks the signup for spam. No email open or click tracking.

Read the full privacy policy ↗