Sitemap Index Files: How to Structure Sitemaps on Large Sites

A sitemap index is an XML file that lists other sitemap files instead of page URLs. You need one as soon as a single sitemap would exceed 50,000 URLs or 50 MB uncompressed, and you will want one well before that, because splitting sitemaps by page type is the cheapest indexing diagnostic available on a large site. One index can reference up to 50,000 child sitemaps.

The format

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/sitemaps/products-1.xml</loc>
    <lastmod>2026-08-22T04:10:00+00:00</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemaps/products-2.xml</loc>
    <lastmod>2026-08-19T04:10:00+00:00</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemaps/articles.xml</loc>
    <lastmod>2026-08-23T11:42:00+00:00</lastmod>
  </sitemap>
</sitemapindex>

The differences from a normal sitemap are small: the root element is <sitemapindex>, each entry is a <sitemap>, and the only children are <loc> (required) and <lastmod> (optional). Here <lastmod> means the time the child sitemap file itself last changed, not the pages inside it.

Limits at a glance

Item	Limit
URLs per child sitemap	50,000
Size per child sitemap	50 MB uncompressed
Sitemaps per index	50,000
Size of the index file	50 MB uncompressed
Nesting	An index may not list another index
Host scope	Children should be on the same site as the index, unless cross-site ownership is proven in Search Console

The nesting rule is worth repeating because plenty of real sites break it. The protocol does not allow an index inside an index. Some crawlers tolerate it, but you should not rely on that. If one index is not enough, publish several and submit each one.

Search Console accepts up to 500 sitemap index files per property, which is another reason nesting is never necessary.

Split by page type, not by count

The lazy way to split is to fill files to 50,000 and start a new one: sitemap-1.xml, sitemap-2.xml and so on, with products, categories and blog posts mixed. It is valid, and it throws away the main benefit.

Split by template or section instead:

/sitemap.xml                     (the index)
/sitemaps/static.xml             (home, about, pricing, legal)
/sitemaps/categories.xml
/sitemaps/products-1.xml
/sitemaps/products-2.xml
/sitemaps/articles.xml
/sitemaps/locations.xml

Now open the Page indexing report in Search Console and use the filter at the top to select a single sitemap. You get an indexed versus not indexed count for that page type alone. When products sit at 94% indexed and locations sit at 31%, you know where the quality or duplication problem is without crawling anything. On a mixed split, the same problem is diluted across every file and invisible.

To get this per-sitemap filtering, submit the child sitemaps in Search Console in addition to the index. Google will process children from the index either way, but the filter lists sitemaps that have been submitted.

Useful extra splits

• By freshness. Put recently published or updated URLs in a small recent.xml and the long tail in archive files. The small file changes often, the big ones rarely, and the index <lastmod> values show crawlers exactly which file is worth re-fetching.
• By language or country on international sites, matching your hreflang structure.
• By suspected quality. If you publish programmatic pages, isolate them. When a batch is not getting indexed, you can see it and decide what to do with it. See programmatic SEO indexing at scale.

How big should each file be?

Legally, up to 50,000 URLs. Practically, smaller files are easier to generate, fetch and debug. Files of 5,000 to 20,000 URLs work well for most sites. There is no indexing benefit to small files by themselves, so do not create thousands of tiny ones either. Choose a size that gives you meaningful groups.

Keep membership stable

A subtle problem on large sites: if products-1.xml contains "the first 50,000 products ordered by last update", then every product edit reshuffles URLs between files. Every file changes every day, the index lastmod values are all "today", and Google has no way to tell which files contain real changes.

Assign URLs to files with a stable rule instead. Common choices:

• ID ranges: products 1 to 20,000 in file 1, 20,001 to 40,000 in file 2.
• A hash or modulo of the ID.
• Creation month: articles-2026-08.xml.

With stable membership, a file's content only changes when one of its own URLs is added, removed or modified. The index <lastmod> for that file then carries real information. More on getting that field right in lastmod best practices.

Generating large sitemaps without hurting the server

Building a sitemap of two million URLs on request, from a live database, is a good way to time out. Patterns that hold up:

1. Generate offline. A scheduled job writes the files to disk or object storage. The web server only serves static files.
2. Write atomically. Write to products-1.xml.tmp, then rename. Crawlers never see a truncated file.
3. Stream, do not buffer. Write URL by URL. Do not build a 50 MB string in memory.
4. Regenerate only what changed. With stable membership this is simple: rebuild the files whose URLs changed, update their lastmod in the index.
5. Remove dead URLs promptly. When a product is deleted or set to noindex, it should leave the sitemap in the next run.

If you serve gzipped files, make sure they are compressed once. A common misconfiguration is a pre-compressed .xml.gz file that the server compresses again with Content-Encoding: gzip, which some parsers then fail to read. A minimal nginx setup for pre-built files:

location /sitemaps/ {
    root /var/www/static;
    types { application/xml xml; application/gzip gz; }
    gzip off;                 # files ending in .gz are already compressed
    add_header Cache-Control "public, max-age=3600";
}

Submitting and referencing the index

Reference the index in robots.txt:

Sitemap: https://www.example.com/sitemap.xml

Then submit the same URL once in Search Console under Sitemaps, and in Bing Webmaster Tools. You do not need to list child sitemaps in robots.txt when they are in the index. The old Google ping endpoint is gone, so there is no value in calling it after each regeneration. Google re-fetches submitted sitemaps by itself.

If the children live on a different host, for example a CDN domain, the scope rule gets in the way. A sitemap normally only covers URLs on its own host. The options are to verify both hosts in the same Search Console account and submit there, or to reference the sitemap from the robots.txt of the host whose URLs it lists. The simpler fix is to proxy the files so they are served from the main host.

Reading the results

In the Sitemaps report, an index shows as type "Sitemap index", and clicking it lists every child with its own status and discovered URL count. Things to look for:

• A child with "Couldn't fetch". Test that exact URL. Frequent causes are a 404 because the generator renamed files, a timeout on a dynamically generated file, or a firewall rule that only affects some paths.
• Discovered count lower than expected. Compare with your database count. A difference usually means the generator is silently skipping records, or the file is truncated at the size limit.
• Errors on specific lines. Unescaped & in URLs, URLs on the wrong host, or an invalid date in lastmod.

Before submitting, validate the index and its children with the XML Sitemap Validator. To get one flat list of every URL across all children, for example to compare with a crawl or a database export, use the Sitemap URL Extractor.

Common mistakes

Index lists children that no longer exist. After a restructure, old entries stay in the index and return 404. Google reports errors, and you lose trust in the report.

The same URL in several children. Not fatal, but it makes per-sitemap indexing counts misleading. One URL, one file.

Mixing indexable and non-indexable URLs. The rules for what belongs in a sitemap are the same at any scale: 200 status, indexable, self-canonical. The XML sitemap guide has the full list.

Expecting the sitemap to solve crawl prioritisation. On very large sites, Google will not crawl everything just because it is listed. Discovery is solved, demand is not. If a large share of your sitemap URLs stay in "Discovered - currently not indexed", the constraint is how much Google wants to crawl, which is the subject of crawl budget explained.

Forgetting the index when debugging. When someone says "the sitemap is fine", check that they looked at the children and not only at the index file, which will always look tidy.

A well split index costs an afternoon to set up. After that, every indexing question on the site starts with a per-section number instead of a guess.