XML Sitemaps: The Complete Guide for Faster Indexing

An XML sitemap is a file that lists the URLs you want search engines to crawl, optionally with the date each one last changed. It does not force indexing and it does not improve rankings. What it does is make discovery reliable: Google and Bing learn about every canonical URL on your site without depending on finding a link to it first. For new sites, large sites and sites with weak internal linking, that is often the difference between a page being found in a day and being found in a month.

This guide covers the format, the limits, what to include, how to submit, and how to read the results.

What a sitemap actually changes

Google finds pages in two main ways: by following links and by reading sitemaps. Links carry more context (anchor text, the importance of the linking page), but they fail in predictable situations:

• The site is new and has no external links, so Googlebot visits rarely.
• The page sits deep in pagination or behind a search form.
• The page is an orphan with no internal links at all (see orphan pages).
• The site has tens of thousands of URLs and Googlebot has not worked through the link graph yet.

A sitemap fixes discovery in all four cases. It does not fix quality. If a page is thin or duplicated, listing it changes nothing about whether Google wants it. That is why you will see sitemap URLs reported as "Discovered - currently not indexed": Google knows the URL exists, and has decided it is not a priority to fetch yet. The article on discovered, currently not indexed explains what to do there.

Sitemaps also give Google a canonicalization hint. Google's documentation lists sitemap inclusion as a weak signal for which URL of a duplicate set you prefer. Weak, but free.

The format

A minimal valid sitemap:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/</loc>
    <lastmod>2026-08-14</lastmod>
  </url>
  <url>
    <loc>https://www.example.com/pricing</loc>
    <lastmod>2026-07-02T09:30:00+00:00</lastmod>
  </url>
</urlset>

Rules that matter in practice:

Rule	Detail
Encoding	UTF-8
Namespace	http://www.sitemaps.org/schemas/sitemap/0.9 on the urlset element
URLs	Absolute, including protocol and host. /pricing is not valid.
URL length	Under 2,048 characters
Escaping	& becomes &, plus ', ", >, <
Max URLs per file	50,000
Max file size	50 MB uncompressed (52,428,800 bytes)
Compression	gzip is allowed (sitemap.xml.gz). The limit applies to the uncompressed size.

The escaping rule catches people constantly. A URL such as https://www.example.com/list?cat=shoes&page=2 must be written as https://www.example.com/list?cat=shoes&page=2. An unescaped ampersand makes the whole file invalid XML, and Search Console will report a parsing error for the entire sitemap, not just that line.

The optional tags

The protocol defines three optional children of <url>:

• <lastmod>: the date the page content last changed, in W3C Datetime format (2026-08-14 or a full timestamp with a timezone).
• <changefreq>: a hint such as daily or monthly.
• <priority>: a value from 0.0 to 1.0.

Google's documentation is explicit: it ignores <priority> and <changefreq>. It uses <lastmod> when the value is "consistently and verifiably accurate". So skip the first two and put your effort into honest lastmod values. There is a full article on lastmod best practices because it is the one field that can help or hurt.

Other formats

Google also accepts RSS 2.0 and Atom feeds, and a plain text file with one URL per line (UTF-8, same 50,000 URL and 50 MB limits). A text sitemap is a perfectly good choice for a small static site:

https://www.example.com/
https://www.example.com/pricing
https://www.example.com/blog/first-post

You lose lastmod, but you also lose every way to break XML.

Extensions

The sitemap namespace can be extended for images, video, news and hreflang. The hreflang version is the most widely useful because it keeps language annotations out of your HTML:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
        xmlns:xhtml="http://www.w3.org/1999/xhtml">
  <url>
    <loc>https://www.example.com/en/</loc>
    <xhtml:link rel="alternate" hreflang="en" href="https://www.example.com/en/"/>
    <xhtml:link rel="alternate" hreflang="de" href="https://www.example.com/de/"/>
  </url>
</urlset>

News sitemaps have their own constraints: only articles published in the last two days, and no more than 1,000 URLs per news sitemap.

Which URLs belong in a sitemap

The rule is short: list every URL you want indexed, in its canonical form, and nothing else. A URL qualifies when all of these are true:

1. It returns 200.
2. It is not blocked by robots.txt.
3. It has no noindex.
4. Its canonical tag points to itself.
5. It is the exact URL format your site links to (same protocol, same host, same trailing slash policy).

Leave out:

• Redirecting URLs and anything returning 4xx or 5xx.
• Parameter variants, session IDs, tracking parameters, sort orders.
• Paginated pages beyond what you actually want in the index.
• Internal search results, cart, checkout, account pages.
• Pages canonicalised to another URL.

Why be strict? Because a sitemap is a statement of intent. When a third of the URLs in it redirect, 404 or carry noindex, the file stops being a useful signal, and your Search Console reports fill with noise (submitted URLs showing up under "Excluded by 'noindex' tag", "Not found (404)" and "Page with redirect") that hides real problems. A clean sitemap also makes the Page indexing report far easier to read, because you can filter it by "All submitted pages" and see exactly how many of the pages you care about are indexed.

A quick way to audit an existing file: pull the URLs out with the Sitemap URL Extractor, run a sample through the Bulk HTTP Status Checker, and look for anything that is not a plain 200.

Generating the file

How you build it matters less than keeping it correct automatically.

• WordPress: core generates /wp-sitemap.xml since version 5.5. SEO plugins such as Yoast and Rank Math replace it with their own index (commonly /sitemap_index.xml) and exclude noindexed content for you.
• Shopify, Wix, Squarespace, Webflow: the platform generates /sitemap.xml and you cannot edit it directly. Control what appears by controlling what is published and indexable.
• Frameworks (Next.js, Nuxt, Astro, Hugo and similar): use the built-in or official sitemap feature and feed it from the same data source that generates your routes, so a new page cannot exist without appearing in the sitemap.
• Custom applications: generate from the database on a schedule or on publish. Write to a temporary file and rename it into place so crawlers never fetch a half-written file.
• Small static sites: a hand-made file is fine. The XML Sitemap Generator turns a URL list into a valid file.

Avoid generators that crawl your site from outside and produce a one-off file. The result is stale the day after you upload it, and a stale sitemap is how new pages get missed.

Once a site goes past 50,000 URLs, or you want per-section reporting, split the sitemap and use an index file. That is covered in sitemap index files for large sites.

Where to put it and the scope rule

Serve the sitemap from the root of the host it describes: https://www.example.com/sitemap.xml. The protocol says a sitemap only applies to URLs at or below its own path. A file at https://www.example.com/blog/sitemap.xml can list /blog/ URLs but not /pricing. It also applies to a single protocol and host, so a sitemap on https://www.example.com should not list https://shop.example.com URLs.

There are two ways around the scope rule when you need one: submit the sitemap in a Search Console account that has verified every host involved, or reference the sitemap from the robots.txt of the host whose URLs it lists.

The file must be fetchable by crawlers. Check that it returns 200 with an XML content type, that it is not blocked in robots.txt, and that no login, firewall rule or bot protection stands in front of it. A sitemap that returns an HTML challenge page to Googlebot is a common cause of "Couldn't fetch" in Search Console.

Submitting it

There are three supported routes, and you can use all of them.

1. robots.txt

Add a line anywhere in your robots.txt:

Sitemap: https://www.example.com/sitemap.xml

The URL must be absolute. You can list several sitemaps on separate lines. This line is independent of any User-agent group, and every crawler that reads robots.txt (Google, Bing and others) picks it up. See the robots.txt guide for the rest of that file.

2. Search Console

Open the property, go to Sitemaps, enter the sitemap URL and submit. This is the route that gives you feedback: fetch status, last read date, discovered URL count and parsing errors. Submitting once is enough. Google re-fetches the file on its own schedule afterwards. Resubmitting the same URL daily does not speed anything up.

3. The Search Console API

sitemaps.submit does the same thing programmatically, which is useful when you manage many properties.

What no longer works: the old ping URL (google.com/ping?sitemap=). Google deprecated it in 2023 and it now returns 404. Bing removed anonymous sitemap pings as well. If a plugin or script still calls those endpoints, it is doing nothing.

For Bing, submit the sitemap in Bing Webmaster Tools under Sitemaps, or rely on the robots.txt line. For quick notification of individual changes, Bing prefers IndexNow (see the IndexNow guide).

Reading the Sitemaps report

After Google processes the file, the report shows a status and a count.

Status	What it means	What to check
Success	File fetched and parsed	Compare "Discovered pages" with the number you expected
Couldn't fetch	Google could not retrieve the file	4xx or 5xx on the sitemap URL, robots.txt block, firewall, or simply not processed yet on a fresh submission
Has errors	Parsed with problems	Open the details: invalid URL, unescaped character, wrong namespace, URL outside the sitemap's scope, oversize file

"Couldn't fetch" right after submitting is frequently a temporary state that clears on its own after the first real fetch. If it persists for days, test the URL with the URL Inspection tool's live test, which shows you the response Googlebot actually receives.

The count in the Sitemaps report is discovered URLs, not indexed URLs. For indexing status, open the Page indexing report and switch the filter at the top to the specific sitemap. That view tells you how many submitted URLs are indexed and which reasons apply to the rest. The page indexing report guide walks through each reason.

Mistakes that make a sitemap useless

Listing non-canonical URLs. HTTP URLs on an HTTPS site, non-www URLs on a www site, or URLs with a trailing slash when the site redirects to the version without. Every one of these produces a "Page with redirect" or duplicate entry.

Setting every lastmod to today. Regenerating the file nightly with the current date on every URL teaches Google that your lastmod values mean nothing, and it stops using them.

Blocking the sitemap or its URLs in robots.txt. Search Console then lists the affected submitted URLs under "Blocked by robots.txt".

Forgetting to update it. A sitemap generated once during launch and never again is worse than none, because it gives you false confidence.

Staging URLs. Sitemaps copied from a staging environment with staging.example.com hosts in every <loc>. Validate after each deployment that the host is right.

Huge single files. A 49,000 URL file is legal but painful. It is slow to generate, slow to fetch and gives you one lump in reporting. Files of a few thousand to ten thousand URLs, grouped by page type, are much easier to debug.

Treating it as a fix for weak internal linking. A page that exists only in the sitemap, with no links from the rest of the site, tells Google that even you do not consider it worth linking to. Sitemaps complement internal linking. They do not replace it.

A short checklist

1. Run the file through the XML Sitemap Validator to confirm format, size and URL count.
2. Spot check that listed URLs return 200, are indexable and self-canonical.
3. Reference the sitemap in robots.txt.
4. Submit it once in Search Console and in Bing Webmaster Tools.
5. Confirm it regenerates automatically when content is published, changed or removed.
6. Review the Page indexing report, filtered by sitemap, every few weeks.

If you would rather not watch this by hand, TurboIndex reads the sitemaps of your Search Console properties, detects new and changed URLs from the file, and tracks the real index status of each one through the URL Inspection API. It is free for one site. Either way, the principle is the same: the sitemap is your list of pages that deserve to be indexed, so keep it accurate and let the reports tell you where Google disagrees.