Meta Robots Tag: Every Directive Google Supports

The robots meta tag is one line in the <head> of a page that tells search engines what they may do with it: whether to index it, whether to follow its links, how much of it to show in results, and when to stop showing it. The default, with no tag at all, is "index everything, follow everything, show a normal snippet". Every directive is an exception to that default. This is a reference for the whole tag; the noindex guide goes deeper on the one directive people use most.

The syntax

<meta name="robots" content="noindex, nofollow">

• name is the crawler the tag addresses: robots for all of them, googlebot for Google, googlebot-news for Google News, bingbot for Bing.
• content holds one or more directives separated by commas.
• Names and directives are case insensitive. Whitespace around the commas does not matter.
• Several tags are allowed. Google reads all of them and combines the directives.
• The tag must be inside <head>. Googlebot stops looking once the body starts.

Valid alternatives to the one-line form:

<meta name="robots" content="noindex">
<meta name="robots" content="nofollow">

<meta name="googlebot" content="noindex">
<meta name="bingbot" content="index">

The second pair keeps a page out of Google while leaving Bing alone.

Every directive Google supports

Directive	Effect	Notes
all	No restrictions	The default. Same as writing nothing
noindex	Do not show the page in results	Only works if the page can be crawled
nofollow	Do not follow links on the page	Google treats nofollow as a hint since 2020
none	Same as noindex, nofollow	Shorthand
noarchive	Do not show a cached copy	Google removed cached links from results in 2024, so it changes nothing visible there; Bing still honours it
nositelinkssearchbox	Do not show a sitelinks search box	Google stopped showing the search box in late 2024, so this is mostly historical
nosnippet	Show no text snippet and no video preview	The title still appears. For part of a page, use the data-nosnippet HTML attribute instead
indexifembedded	Allow Google to index the page's content when it is embedded in another page through an iframe or similar, even though the page itself is noindex	Only has an effect together with noindex
max-snippet:[number]	Limit the text snippet to that many characters	0 means no snippet, -1 means no limit
max-image-preview:[setting]	Size of image previews	none, standard or large
max-video-preview:[number]	Maximum seconds of a video preview	0 means a still image only, -1 means no limit
notranslate	Do not offer a translation of the page in results
noimageindex	Do not index images on the page	Images may still be indexed if other pages embed them
unavailable_after: [date/time]	Stop showing the page after that moment	Date in RFC 822 or ISO 8601 format. Google treats it like noindex after the date

Examples of the parameterised ones:

<meta name="robots" content="max-snippet:160, max-image-preview:large, max-video-preview:-1">
<meta name="robots" content="unavailable_after: 2026-12-31T23:59:00+00:00">
<meta name="robots" content="noindex, indexifembedded">

Google ignores directives it does not know. Values from other engines or old tools (noodp, noydir, nocache) do nothing for Google and are silently skipped.

data-nosnippet, the attribute cousin

Not a meta directive but part of the same specification: wrapping an element with data-nosnippet keeps that text out of snippets while the rest of the page stays eligible.

<p>The public part of the article.</p>
<p data-nosnippet>Member pricing: 42 per seat.</p>

It works on span, div and section elements in the HTML source. Text added by JavaScript inside such an element is still covered.

Defaults: what index, follow really does

index and follow are implied on every page. A page with no tag, a page with content="all", and a page with content="index, follow" are treated identically. The tag that many CMS plugins print by default looks like this:

<meta name="robots" content="index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1">

Everything in it is the default anyway, except that max-image-preview:large opts the page in to large image previews, which Google uses in some result layouts and in Discover. WordPress itself has printed max-image-preview:large since version 5.7 for public pages. There is no SEO benefit in adding index, follow by hand, and no harm either.

Engine-specific names

Name	Read by
robots	All crawlers
googlebot	Google Search and the Google crawlers that respect robots rules
googlebot-news	Google News only
bingbot	Bing (and the engines that use Bing's index)
yandex	Yandex

Bing reads the same core set (noindex, nofollow, noarchive, nosnippet, the max-* family) and adds nocache, which it documents together with noarchive as a control over how its Copilot answers may use a page. Google does not support nocache.

When you need a rule for one engine, use the specific name. googlebot: noindex in a tag with name="googlebot" keeps a page out of Google while Bing and others can still index it. That is sometimes useful for pages that one engine handles badly (for example a parameter variant that Google canonicalises correctly and Bing does not), and rarely otherwise.

Conflicts and precedence

Google's rule is simple: the most restrictive directive wins, across every place a directive can appear.

• index and noindex on the same page: not indexed.
• max-snippet:100 and nosnippet: no snippet.
• <meta name="robots" content="index"> plus <meta name="googlebot" content="noindex">: not indexed by Google.
• A meta tag saying index and an X-Robots-Tag: noindex header: not indexed.

This matters in practice because two systems often write tags independently: the theme prints one, an SEO plugin prints another, and a CDN adds a header. Each one looks fine alone and the combination is noindex. Always check the complete response, not the template.

One combination that is not a conflict in the formal sense but is a mistake in practice: noindex together with a rel="canonical" pointing elsewhere. Google has described it as a mixed signal: noindex says "this page should not be in the index", canonical says "this page is a copy of that one, merge them", and Google may act on either. Pick one signal per page: canonical for duplicates, noindex for pages that should not exist in results at all. The decision guide is in noindex vs robots.txt vs canonical.

The header equivalent

Every directive above also works in the X-Robots-Tag HTTP header, which is the only option for files without an HTML head.

X-Robots-Tag: noindex, nofollow
X-Robots-Tag: googlebot: noindex
X-Robots-Tag: unavailable_after: 31 Dec 2026 23:59:00 GMT

Several headers in one response are combined, and headers combine with meta tags the same way (most restrictive wins). Server configuration for nginx, Apache and Next.js is in the noindex guide.

Setting the tag in WordPress and Shopify

WordPress. The core Settings > Reading > Discourage search engines option prints noindex site wide. Yoast and Rank Math both print a full robots tag on every page and let you change it per content type (Yoast: Settings > Content types; Rank Math: Titles & Meta) and per post (Yoast: the Advanced section of the sidebar; Rank Math: the Advanced tab, Robots Meta). With a plugin active, do not also add a tag in the theme; you will end up with two.

Shopify. There is no per page checkbox in the admin for the robots tag. The documented route is a seo.hidden metafield: add a metafield with namespace seo, key hidden, type integer, value 1 to a product, collection, page or article, and Shopify adds a noindex tag and drops the URL from the sitemap. For anything else, edit layout/theme.liquid under Online Store > Themes > Edit code and add a condition inside the head:

{% if template contains 'search' or template contains 'customers' %}
  <meta name="robots" content="noindex">
{% endif %}

Shopify's generated robots.txt already disallows cart, checkout, account and search paths, so the meta tag there is a belt-and-braces measure.

Common mistakes

• Tag in the body. Page builders that let you paste "custom HTML" sometimes place it after <body>. Google ignores it.
• Tag inside an HTML comment or a template that is commented out. Visible in the source, not in effect.
• Two plugins, two tags. One says index, the other says noindex. Noindex wins.
• Relying on JavaScript to remove noindex. When the raw HTML carries noindex, Google does not render the page, so the script never runs for Google.
• Noindex on paginated category pages. Products reachable only through page 2 onwards lose their internal links once those pages are dropped. Leave pagination indexable, or link the products from elsewhere.
• Noindex combined with a robots.txt block. The block prevents the tag from being read. Covered in Indexed, though blocked by robots.txt.
• Expecting instant effect. Google applies the tag at the next crawl, which can be days or weeks away for a deep page.

How to test

1. View the source of the page (not the rendered DOM) and search for name="robots" and name="googlebot".
2. Check the headers. curl -I https://www.yoursite.com/page and look for X-Robots-Tag. CDNs and reverse proxies add headers the application never sees.
3. Run the URL through the meta tags analyzer, which lists every meta tag in the head with its value, or the indexability checker, which reads the meta tag, the header, the status code and the canonical together and gives a yes or no.
4. Ask Google. In Search Console, inspect the URL and read Indexing allowed? under Page indexing. For a page you just changed, run Test live URL and look at the rendered HTML, which catches tags added or changed by scripts.

If you are writing the tag for a new site, the meta tag generator produces the head block with title, description and robots values filled in, which avoids typos in the directive names.

A final practical note: whatever a page's tag says, it is pointless to submit that page for indexing if the tag says noindex. TurboIndexBot, the crawler behind TurboIndex's pre-submission check, reads both the meta tag and the X-Robots-Tag header before a URL is sent to Google and holds back any page that carries noindex or none, so the directive you set in the template is the one that decides.