Noindex vs Robots.txt vs Canonical: Which One to Use When

These three tools are constantly mixed up because they all make pages "not show up". They act at different stages. Robots.txt controls whether a crawler may fetch a URL. Noindex controls whether a fetched page may be stored in the index. A canonical tag tells Google which of several similar URLs should represent them all. Choosing the wrong one, or stacking them, is a common cause of pages that refuse to leave the index and pages that vanish by accident.

What each one does

	robots.txt disallow	noindex	rel=canonical
Acts on	Crawling	Indexing	Duplicate consolidation
Type	Rule that Google obeys	Rule that Google obeys	Hint that Google weighs
Where it lives	/robots.txt on the host	Meta tag in <head> or X-Robots-Tag header	Link element in <head>, Link header
Page must be crawlable for it to work?	Not applicable	Yes	Yes
Removes the URL from search results?	No. The URL can be indexed without content	Yes, after the next crawl	Usually, by folding it into the canonical URL
Consolidates link signals?	No	No	Yes
Saves crawl requests?	Yes	No, the page must be fetched	No, duplicates are still crawled, though less often
Works for non-HTML files?	Yes	Yes, via the header	Yes, via the header
Typical use	Infinite filter URLs, internal search, cart	Thank you pages, thin archives, internal tools pages, staging leaks	Tracking parameters, sort orders, print versions, syndication

Two rows deserve a second look.

"Page must be crawlable" is the row that causes the most damage. Noindex and canonical are both instructions written on the page. If Googlebot is not allowed to open the page, it cannot read them.

"Removes the URL from search results" for robots.txt is "no". A blocked URL with links pointing at it can be indexed as a bare URL, reported in Search Console as "Indexed, though blocked by robots.txt".

The syntax, for reference

Robots.txt:

User-agent: *
Disallow: /internal-search/

Noindex as a meta tag:

<meta name="robots" content="noindex">

Noindex as an HTTP header, which is the only option for PDFs, images and other non-HTML files:

X-Robots-Tag: noindex

location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex";
}

<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

Canonical:

<link rel="canonical" href="https://www.example.com/shoes">

You can see which of these a URL currently carries, including header versions, with the Indexability Checker or the HTTP Headers Checker.

The robots.txt plus noindex trap

The sequence goes like this. Someone wants a set of pages out of Google. They add a noindex tag. To be thorough, they also add a disallow rule in robots.txt. Weeks later the pages are still in the index.

What happened: Googlebot checks robots.txt before fetching. The disallow rule tells it not to fetch. It never downloads the HTML, so it never sees the noindex. The pages that were already indexed stay indexed, often with the note that no information is available for the page.

The fix is to do things in order:

1. Remove the disallow rule so the pages can be crawled.
2. Keep the noindex in place.
3. Wait for Google to recrawl. You can speed up important URLs with URL Inspection, and a sitemap listing the noindexed URLs with a fresh lastmod helps them get recrawled. Remove that sitemap once they have dropped.
4. Confirm with URL Inspection that the pages report "Excluded by 'noindex' tag".
5. Optionally re-add the disallow afterwards, if crawl volume on those URLs is a real problem.

The same logic applies to two relatives of the trap:

• Robots.txt plus canonical. Blocking duplicate URLs stops Google from seeing that they are duplicates and from consolidating their signals.
• Robots.txt plus 404 or 301. Blocking removed or moved URLs stops Google from seeing that they are gone or where they went.

Also gone for good: a Noindex: line inside robots.txt. It was never documented, and Google stopped honouring it in 2019.

Which to use when

Start from the outcome you want.

"This page must never appear in Google, but users need it"

Use noindex, and leave the page crawlable. Examples: order confirmation pages, internal search results you link to, tag archives with no unique value, landing pages for paid campaigns, author pages you consider thin.

"These URLs are copies of another page"

Use a canonical, or a 301 redirect if users never need the duplicate URL. Examples: URLs with tracking parameters, ?sort= variants, print versions, HTTP and non-www variants (redirect these), the same product reachable through several category paths.

Do not use noindex here. Google's documentation advises against noindex for canonicalization, because it drops the page without transferring anything to the version you prefer.

"Crawlers are wasting time in a huge URL space"

Use robots.txt. Examples: faceted navigation that can generate millions of combinations, calendar pages reaching into the year 2090, internal search with arbitrary queries, session ID URLs. Google's crawl budget guidance recommends robots.txt for this, and warns against using noindex as a crawl control because each page still has to be requested. Accept the side effect: a few of those URLs may be indexed without content if they are linked from elsewhere. They almost never rank for anything. Background in crawl budget explained.

"This page no longer exists"

Use none of the three. Return 404 or 410, or a 301 to the closest replacement. Status codes are the clearest removal signal there is. See the HTTP status codes guide.

"This content is private"

Use none of the three. Robots.txt is a public file that lists the paths you would like hidden, and noindex only affects search engines that respect it. Put private content behind authentication.

"I need it out of Google today"

Use the Removals tool in Search Console to hide the URL quickly (the block lasts about six months), and apply the permanent measure in parallel: noindex, a 404 or 410, or a login. The Removals tool alone is temporary.

Quick reference

Situation	Use
Thank you or confirmation page	noindex
Internal search results	robots.txt (or noindex if the volume is small)
UTM and click ID parameters	canonical
Sort order parameters	canonical
Millions of filter combinations	robots.txt
HTTP, non-www, trailing slash variants	301 redirect
PDF that duplicates an HTML page	canonical via Link header
PDF that should not be found in search	X-Robots-Tag: noindex
Staging site	HTTP authentication
Deleted product with no replacement	404 or 410
Deleted product with a successor	301
Syndicated copy on a partner site	noindex on the partner copy, or a cross-domain canonical

Combining signals: what conflicts and what does not

noindex plus canonical to another URL: conflict. One says "this equals that page", the other says "drop this page". Google's systems will try to guess what you meant, and you cannot predict which way the guess goes. Pick one.

noindex plus self-referencing canonical: fine. Many CMS templates output both. Noindex wins and the page is dropped.

noindex plus nofollow: rarely needed. noindex, nofollow also tells Google not to follow the links on the page. Unless you have a specific reason, plain noindex is enough.

noindex in the sitemap: contradiction. A sitemap lists URLs you want indexed. Listing noindexed URLs produces warnings in Search Console. The only exception is the temporary sitemap used to speed up removal, described above.

Canonical target that is noindexed, blocked or redirected: broken. The canonical target must be a 200, crawlable, indexable page.

Noindex added with JavaScript: unreliable in one direction. Google can see a noindex that JavaScript adds. The reverse does not work: if the raw HTML contains noindex and JavaScript removes it, Google may skip rendering when it sees the tag and never notice the removal. Details in JavaScript SEO.

How long each takes to show an effect

None of them is instant, because each depends on a crawl.

• Robots.txt: Google generally refreshes its cached copy within 24 hours. Crawling of the blocked paths stops once the new file is picked up.
• Noindex: takes effect the next time Google crawls that URL. For a frequently crawled page that can be days. For a deep page it can be months. You can request a recrawl with URL Inspection.
• Canonical: applied when Google has crawled both the duplicate and the target and re-evaluated the group. Expect weeks on larger sites.

Check progress in the Page indexing report, where these show as Excluded by 'noindex' tag, "Blocked by robots.txt" and "Alternate page with proper canonical tag". All three are, in the right context, signs that your setup works and not errors to eliminate.

A short checklist before you ship

1. Name the outcome: not crawled, not indexed, or consolidated.
2. Choose the single mechanism that matches it.
3. Make sure nothing blocks Google from reading that mechanism.
4. Remove contradictions: sitemap entries, internal links and hreflang should agree with your choice.
5. Test a sample of URLs after deployment, then check Search Console a few weeks later.

If you need more depth on any one of them, see the robots.txt guide and the canonical tags guide.