Can I use robots.txt to remove pages from Google?

No. Robots.txt stops crawling, not indexing. A blocked URL that other pages link to can still be indexed with no content. To remove a page, allow crawling and add a noindex meta tag or header, or return 404 or 410, or use the Removals tool for urgent cases.

How do I block Googlebot only?

Add a group with User-agent: Googlebot and Disallow: /. Googlebot follows the most specific group that matches it and ignores the User-agent: * group, so any rules you want Googlebot to follow must be repeated in its own group.

How do I block AI crawlers but keep Google Search?

Add a group listing the tokens you want to block (GPTBot, ClaudeBot, CCBot and so on) with Disallow: /, and leave Googlebot unblocked. Google-Extended is a separate token that controls Gemini training and grounding use; blocking it does not affect Search.

I accidentally shipped Disallow: / to production. What now?

Fix the file, then open the robots.txt report in Search Console (Settings, Crawling, robots.txt) and request a recrawl. Google caches robots.txt for up to about 24 hours, so the block clears quickly. Pages that were already indexed usually stay indexed; request indexing for the important ones that dropped.

Robots.txt to Block All Crawlers, One Bot or AI Bots

To block every crawler, serve this as https://www.yoursite.com/robots.txt:

User-agent: *
Disallow: /

That is the whole file. Every crawler that respects robots.txt (Googlebot, Bingbot, the AI crawlers, SEO tools) stops fetching pages on that host. It does not remove pages from Google, it does not stop crawlers that ignore the file, and it does not protect anything private. The rest of this article gives the exact files for the other common cases, explains what the block does and does not do, and shows how to recover when the block was a mistake.

How a crawler reads the file

Three rules explain almost every surprise with robots.txt:

One file per host. www.yoursite.com, yoursite.com, blog.yoursite.com and the http versions each have their own robots.txt. A rule on one host says nothing about the others.
A crawler follows one group only: the group with the most specific User-agent that matches it. If there is a Googlebot group, Googlebot reads that group and ignores User-agent: * entirely.
The file controls fetching, nothing else. Indexing, snippets and link following are controlled by the robots meta tag and the X-Robots-Tag header, which a crawler can only read if it is allowed to fetch the page.

Syntax, wildcards and precedence between Allow and Disallow are in the robots.txt guide. The rest of this page is the recipes.

Block one bot

User-agent: Googlebot
Disallow: /

User-agent: *
Disallow:

Googlebot is blocked from everything, every other crawler is allowed everywhere (an empty Disallow: means "nothing is disallowed"). Because of rule 2 above, a Googlebot group has to contain every rule you want Googlebot to follow; it inherits nothing from *.

Common tokens:

Token	Crawler
`Googlebot`	Google Search (smartphone and desktop)
`Googlebot-Image`	Google Images. Falls back to the `Googlebot` group when it has no group of its own
`Bingbot`	Microsoft Bing
`Yandex`	Yandex
`DuckDuckBot`	DuckDuckGo's own crawler (its results mostly come from Bing)
`Baiduspider`	Baidu
`AhrefsBot`, `SemrushBot`	SEO tool crawlers
`TurboIndexBot`	The crawler TurboIndex uses to check URLs before submission

Block everyone except Google and Bing

User-agent: Googlebot
User-agent: Bingbot
Disallow:

User-agent: *
Disallow: /

Several User-agent lines may share one group. Both named crawlers get the empty Disallow (allowed everywhere); everyone else gets the full block. Google's other crawlers that have no group of their own, such as Googlebot-Image, use the Googlebot group, so this file keeps them working too.

Block AI crawlers while keeping Googlebot

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

User-agent: *
Disallow:

What each token controls:

GPTBot is OpenAI's training crawler. OAI-SearchBot and ChatGPT-User are separate tokens for ChatGPT search and user-triggered fetches; block them too if you want no OpenAI traffic at all.
ClaudeBot is Anthropic's crawler. CCBot is Common Crawl, whose archive many models train on.
Google-Extended is not a crawler. It is a control token: Googlebot keeps crawling as usual, and the rule tells Google not to use the fetched content for training Gemini models or for grounding in Gemini apps. Google states that it does not affect inclusion or ranking in Search, and it does not keep pages out of AI Overviews, which are a Search feature.
Applebot-Extended works the same way for Apple's models; Applebot itself keeps crawling for Siri and Spotlight.
Bytespider (ByteDance), PerplexityBot and meta-externalagent (Meta) are the other large ones at the time of writing.

The list changes. Check each company's published crawler documentation before trusting a copy-pasted block from a forum, and remember that a crawler which does not announce itself cannot be blocked by robots.txt at all; that is a job for the firewall.

Block a folder, a file type or a parameter

User-agent: *
Disallow: /admin/
Disallow: /cgi-bin/
Disallow: /*.pdf$
Disallow: /*?sessionid=
Allow: /admin/public/

A path rule matches by prefix: /admin/ blocks everything under it.
* matches any run of characters and $ anchors the end, so /*.pdf$ blocks URLs ending in .pdf.
Allow carves exceptions out of a Disallow; for Google the longer (more specific) rule wins when both match.

Test every rule against real URLs before shipping. The robots.txt tester takes a URL and a user agent and tells you which line matched. For a starting file, the robots.txt generator writes the groups for you.

Why robots.txt does not deindex anything

People search for "robots.txt noindex" because they want a page out of Google and the file looks like the place to say so. It is not, for two reasons.

First, Google dropped support for a Noindex: line in robots.txt in 2019. Writing it does nothing.

Second, a Disallow rule stops Googlebot fetching the page, but Google can still index the URL from links pointing at it, with the anchor text as its title and no description. The Page indexing report calls this "Indexed, though blocked by robots.txt", and the result in Google looks broken: a bare URL with the note that no information is available. Worse, if the page carries a noindex tag, Google cannot see it, because it is not allowed to fetch the page.

The correct tools for removing a page:

Goal	Tool
Keep a page out of results, keep it live for users	`noindex` meta tag or header, page allowed in robots.txt
Page no longer exists	Return 404 or 410
Content is private	Password or HTTP authentication (401)
It has to disappear this week	Removals tool in Search Console (temporary, about six months) plus one of the above
Save crawl capacity on a huge URL space nobody should see	robots.txt, accepting that stray URLs may be indexed without content

The meta tag and header syntax is in the noindex guide, and the state you get when robots.txt is used by mistake is explained in Indexed, though blocked by robots.txt.

Staging sites: do not rely on Disallow: /

A staging copy with Disallow: / has three problems. Links to it (from a shared preview, a ticket, a tweet) can still get its URLs indexed without content. Any crawler that ignores robots.txt reads everything. And the file travels with the code: the most common way a production site ends up fully blocked is a staging robots.txt deployed by mistake.

Better options:

HTTP authentication on the whole staging host. Crawlers get 401 and index nothing. Google reports blocked pages as "Blocked due to unauthorized request (401)", which is the outcome you want.
An X-Robots-Tag: noindex header set at the staging web server, outside the application code, so it cannot be deployed to production by accident.

The Disallow: / accident on launch

Symptoms: traffic falls off within days of a deploy, the Page indexing report shows "Blocked by robots.txt" climbing, and URL Inspection reports "Crawl allowed? No: blocked by robots.txt" for everything.

Recovery:

Fix the file on the live host and confirm with curl https://www.yoursite.com/robots.txt that the fix is what the server sends (not a cached old copy at the CDN).
In Search Console open Settings > Crawling > robots.txt > Open report. The report shows the version Google last fetched and when. Use the request a recrawl option so Google refetches it now rather than at the next scheduled fetch. Google generally caches robots.txt for up to 24 hours, so even without the request the block lifts within a day.
Open the "Blocked by robots.txt" row in the Page indexing report and click Validate fix once the new file is fetched.
Request indexing for the handful of pages that matter most.

Two things soften the damage. Pages that were already indexed when the block went live usually stay indexed; Google keeps them with their old content until it can crawl again. And a block of a few days rarely changes rankings on its own. The pages that suffer are new ones published during the block, which were never fetched. The complete repair sequence, including finding the rule when it is less obvious than Disallow: /, is in Blocked by robots.txt: how to fix it.

Testing the file

Fetch it with curl from outside your network and read what comes back. A 404 means "allow everything" to Google; a 5xx makes Google treat the whole site as disallowed while it keeps retrying, and after a long outage it falls back to the last good copy it has or, failing that, to no restrictions.
Test individual URLs per user agent with the robots.txt tester.
Check the Search Console robots.txt report after a change to see the version Google is using.
Repeat for every host: the www and non-www versions and each subdomain have separate files.

TurboIndexBot rules

TurboIndexBot fetches robots.txt, sitemaps and each sitemap URL once before the URL is submitted, so that a 404, a redirect or a noindex page is never sent. It obeys rules addressed to it by name and re-reads them every ten minutes:

User-agent: TurboIndexBot
Disallow: /

It does not apply the generic User-agent: * group, because each request is a check that a person asked for rather than a crawl, which means a site that blocks all crawlers can still be checked by its own owner. Blocking it by name on a site connected to TurboIndex stops the pre-check and the sitemap reads, so new pages are no longer picked up. The user agent string, request behaviour and contact details are on the TurboIndexBot page.

What to do next

Decide what you actually want: no crawling at all (rare), one bot kept out, AI training opted out, or a page removed from results (not a robots.txt job).
Write the smallest file that does that, one group per crawler you treat differently.
Test three real URLs per user agent before deploying.
Keep staging protected with authentication, not with the robots.txt that might get deployed.
After any change, check the robots.txt report in Search Console the next day.

Frequently asked questions

What robots.txt blocks all crawlers?: Two lines: User-agent: * followed by Disallow: /. Every crawler that honours robots.txt stops fetching pages on that host. It does not remove pages that are already indexed and it does not stop crawlers that ignore the file.
Can I use robots.txt to remove pages from Google?: No. Robots.txt stops crawling, not indexing. A blocked URL that other pages link to can still be indexed with no content. To remove a page, allow crawling and add a noindex meta tag or header, or return 404 or 410, or use the Removals tool for urgent cases.
How do I block Googlebot only?: Add a group with User-agent: Googlebot and Disallow: /. Googlebot follows the most specific group that matches it and ignores the User-agent: * group, so any rules you want Googlebot to follow must be repeated in its own group.
How do I block AI crawlers but keep Google Search?: Add a group listing the tokens you want to block (GPTBot, ClaudeBot, CCBot and so on) with Disallow: /, and leave Googlebot unblocked. Google-Extended is a separate token that controls Gemini training and grounding use; blocking it does not affect Search.
I accidentally shipped Disallow: / to production. What now?: Fix the file, then open the robots.txt report in Search Console (Settings, Crawling, robots.txt) and request a recrawl. Google caches robots.txt for up to about 24 hours, so the block clears quickly. Pages that were already indexed usually stay indexed; request indexing for the important ones that dropped.