Noindex: The Meta Tag and X-Robots-Tag Header Guide

noindex is a directive that tells search engines not to include a page in their results. You give it in one of two equivalent forms: a <meta name="robots" content="noindex"> tag in the HTML head, or an X-Robots-Tag: noindex HTTP header. Google acts on it at the next crawl of the page, and it only works if Googlebot is allowed to fetch the page, so a URL blocked in robots.txt cannot be noindexed. This article covers the exact syntax for the common servers and platforms, the variants with follow and nofollow, and how to check that the directive is really there.

The meta tag

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Internal page</title>
  <meta name="robots" content="noindex">
</head>
<body>
  ...
</body>
</html>

Rules Google applies:

• The tag must be in the <head>. A meta robots tag in the body is ignored.
• name="robots" addresses every crawler. name="googlebot" addresses Google only; name="bingbot" addresses Bing only.
• Directives and names are case insensitive. NOINDEX and noindex are the same.
• Several directives go in one content attribute separated by commas (noindex, nofollow), or in several tags. Google combines them.
• The tag can be added by JavaScript during rendering and Google will see it. The reverse is not true: if the raw HTML contains noindex, Google skips rendering, so JavaScript that removes the tag has no effect.

The full set of directives that can share the tag (nosnippet, max-snippet, unavailable_after and the rest) is in the meta robots tag guide.

The X-Robots-Tag header

The header carries the same directives in the HTTP response:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

It exists because files without an HTML head (PDFs, images, video files, XML feeds, plain text) cannot carry a meta tag. It is also convenient when a rule covers a whole directory or file type and you would rather set it once in the server configuration than in every template.

Two forms are valid:

X-Robots-Tag: noindex, nofollow
X-Robots-Tag: googlebot: noindex

The second form targets one crawler. Several X-Robots-Tag headers in one response are allowed and are combined.

nginx

# Every PDF under /downloads/
location ~* ^/downloads/.*\.pdf$ {
    add_header X-Robots-Tag "noindex, nofollow" always;
}

# A whole section, for example an internal search
location /search/ {
    add_header X-Robots-Tag "noindex" always;
}

One nginx detail that catches people: add_header directives are inherited from the parent block only when the current block has none of its own. If your server block sets security headers with add_header and a location block adds X-Robots-Tag, the location block has to repeat the security headers too, or they disappear for that location.

Apache (.htaccess)

<IfModule mod_headers.c>
  # Every PDF and Word file
  <FilesMatch "\.(pdf|docx?)$">
    Header set X-Robots-Tag "noindex, nofollow"
  </FilesMatch>
</IfModule>

For a directory, put the .htaccess file in that directory with just the Header set line (no FilesMatch), and every file served from it gets the header. mod_headers has to be enabled; on shared hosting it almost always is.

Next.js

For an HTML page in the App Router, use the robots field of the metadata export. Next.js renders it as the meta tag:

// app/internal/page.tsx
import type { Metadata } from "next";

export const metadata: Metadata = {
  robots: {
    index: false,
    follow: true,
  },
};

export default function InternalPage() {
  return <main>...</main>;
}

Output in the head:

<meta name="robots" content="noindex, follow" />

When the decision depends on data, return the same object from generateMetadata:

// app/products/[id]/page.tsx
import type { Metadata } from "next";

type Props = { params: Promise<{ id: string }> };

export async function generateMetadata({ params }: Props): Promise<Metadata> {
  const { id } = await params;
  const product = await getProduct(id);
  return {
    title: product.name,
    robots: product.discontinued ? { index: false, follow: true } : undefined,
  };
}

For non-HTML files, or for a whole path, set the header in next.config.js:

// next.config.js
module.exports = {
  async headers() {
    return [
      {
        source: "/drafts/:path*",
        headers: [{ key: "X-Robots-Tag", value: "noindex" }],
      },
    ];
  },
};

WordPress

Site wide: Settings > Reading > Discourage search engines from indexing this site. This outputs a noindex meta tag on every page. It is the single most common reason a WordPress site is missing from Google, because it is often switched on during development and forgotten.

Per post or page with Yoast SEO: in the editor, open the Yoast sidebar, expand Advanced, and set Allow search engines to show this Post in search results? to No. For a whole content type, go to Yoast SEO > Settings > Content types, pick the type and switch off Show in search results.

Per post or page with Rank Math: in the Rank Math meta box open the Advanced tab and under Robots Meta tick No Index. For a content type, go to Rank Math > Titles & Meta, pick the type and set its robots meta.

Without a plugin, WordPress has had the wp_robots filter since version 5.7:

<?php
// In a small plugin or the theme's functions.php
add_filter( 'wp_robots', function ( array $robots ): array {
    if ( is_tag() || is_author() || is_search() ) {
        $robots['noindex'] = true;
        $robots['follow']  = true;
    }
    return $robots;
} );

WordPress merges this into the <meta name='robots' ...> tag it already prints, so you do not get two tags. For the header form, send it before the template renders:

<?php
add_action( 'template_redirect', function () {
    if ( is_search() ) {
        header( 'X-Robots-Tag: noindex' );
    }
} );

noindex, nofollow, none and follow

Value	Index the page?	Follow its links?	Typical use
index, follow	Yes	Yes	The default. Writing it changes nothing
noindex	No	Yes (follow is implied)	Utility pages, thin archives, thank you pages
noindex, follow	No	Yes	Same as above, written out
noindex, nofollow	No	No	Pages whose links you do not want crawled either
none	No	No	Shorthand for noindex, nofollow
nofollow	Yes	No	Rarely useful on a whole page

noindex alone and noindex, follow are identical, because follow is the default. Most of the time noindex is what you want: the page stays out of results, and Google can still move through its links to find other pages.

nofollow at page level stops Google from using any link on the page for discovery or ranking, as a hint. It is a blunt tool. For individual links, rel="nofollow" (or rel="sponsored" and rel="ugc") on the anchor is the precise version.

How Google treats noindex over time

At the next crawl after you add the directive, Google drops the page from results. The URL stays known to Google, keeps being crawled now and then, and shows as "Excluded by 'noindex' tag" in the Page indexing report. That is the correct, final state for a page you meant to exclude.

Google has also said that a page which stays noindex for a long time is eventually treated as noindex, nofollow: the links on it stop being followed. So the idea of a noindex hub page that passes authority to the pages it links to does not hold up long term. Keep hub pages indexable, or link to the important pages from somewhere that is.

A noindex page should also leave the sitemap. Listing it there sends Google two contradictory messages and shows up as a "submitted" exclusion in the report.

The robots.txt trap

Noindex is read from the page. If robots.txt says Disallow: /private/, Googlebot never fetches /private/report, never sees the tag, and may still index the URL from links to it, showing it with no description. That state is called "Indexed, though blocked by robots.txt" and it is covered in its own article.

The order of operations for a page you want out of Google:

1. Make sure the URL is allowed in robots.txt.
2. Add noindex.
3. Wait for Google to recrawl it and for the Page indexing report to show it as excluded by noindex.
4. Only then, if you also want to save crawl capacity, block it in robots.txt. Most sites can skip this step.

The three-way comparison of the tools is in noindex vs robots.txt vs canonical.

How to check what a page really says

Browsers and caches hide things. Check at the HTTP level:

curl -s -A "Googlebot" -D - -o /dev/null https://www.example.com/page | grep -i x-robots-tag
curl -s -A "Googlebot" https://www.example.com/page | grep -i '<meta name="robots"'

Or use the free tools: the indexability checker fetches a URL, reads the meta robots tag, the header, the status code and the canonical, and says plainly whether the page can be indexed. The HTTP headers checker shows the raw response headers, which is where an X-Robots-Tag added by a CDN or a plugin tends to hide.

For Google's own view, inspect the URL in Search Console. Under Page indexing, the line Indexing allowed? says "No: 'noindex' detected in 'robots' meta tag" or "'noindex' detected in 'X-Robots-Tag' http header". If the raw HTML looks clean but Google reports noindex, run Test live URL and look at the rendered HTML: something is injecting the tag at render time.

Removing noindex by mistake, or on purpose

To bring a page back:

1. Remove the tag or the header. Check both; sites often have one in the template and another from a plugin or the CDN.
2. Confirm with curl or the indexability checker that neither remains, and that the rendered page is clean too.
3. Inspect the URL and click Request indexing.
4. For many pages, open the "Excluded by 'noindex' tag" row in the Page indexing report and click Validate fix.

Recovery is usually days for a requested URL and up to a few weeks for a large group. Where the stray noindex came from and how to find it in each CMS is the subject of Excluded by 'noindex' tag.

noindex and submissions

A noindex page sent to Google through any submission route is wasted effort: Google fetches it and excludes it. TurboIndexBot, the crawler TurboIndex uses, reads the meta robots tag and the X-Robots-Tag header of every sitemap URL before any submission and never sends a page that carries noindex; the page is listed as blocked with that reason instead, so a template mistake shows up on the first run rather than in the Page indexing report two weeks later. What it fetches and how to control it is on the TurboIndexBot page.

What to do next

1. Decide per page type whether it belongs in search results. Tag archives, internal search, cart, account and thank you pages usually do not.
2. Set noindex in the template or the server config, using the snippets above.
3. Make sure none of those URLs are blocked in robots.txt and none are in the sitemap.
4. Verify with curl or the indexability checker, then with URL Inspection after the next crawl.