Robots.txt is a plain text file at the root of a host that tells crawlers which URLs they may fetch. It controls crawling, not indexing: a disallowed URL can still appear in Google if something links to it, and a noindex tag on a disallowed page will never be seen. Use it to keep crawlers out of URL spaces that waste their time, and use other tools to keep pages out of search results.
Where the file lives and what it covers
The file must be at /robots.txt on the root of the host. It applies to exactly one combination of protocol, host and port:
| robots.txt location | Applies to | Does not apply to |
|---|---|---|
https://www.example.com/robots.txt |
https://www.example.com/* |
https://example.com/, https://shop.example.com/, http://www.example.com/ |
https://example.com/robots.txt |
https://example.com/* |
any subdomain |
Every subdomain needs its own file. A file at /blog/robots.txt is ignored. This matters with CDNs and asset hosts: if your images are served from cdn.example.com, the rules for them live on that host.
The file must be UTF-8 plain text. Google reads at most 500 KiB of it and ignores anything after that limit.
The syntax
Google supports four fields: user-agent, allow, disallow and sitemap. Field names are case insensitive. Path values are case sensitive, so /Admin/ and /admin/ are different rules.
# Everything after a hash is a comment
User-agent: *
Disallow: /cart
Disallow: /search
Allow: /search/help
User-agent: Googlebot-Image
Disallow: /private-images/
Sitemap: https://www.example.com/sitemap.xml
Rules are organised in groups. A group starts with one or more User-agent lines and contains the Allow and Disallow lines that follow. Sitemap lines stand alone and apply to everyone.
How a crawler picks its group
A crawler uses the single group whose user agent matches it most specifically, and ignores the rest. With the file above, Googlebot-Image follows only the second group. It does not also obey the * group. This surprises people: adding a dedicated User-agent: Googlebot group with one rule means Googlebot stops following everything you wrote under *. If you create a specific group, copy the general rules into it.
If the same user agent appears in several groups, Google combines them into one.
How path matching works
Rules match from the start of the URL path (including the query string). Disallow: /cart blocks /cart, /cart/, /cart?step=2 and also /cartoons. To block only the directory, write Disallow: /cart/.
Two wildcards are supported by Google and Bing:
*matches any sequence of characters.$anchors the end of the URL.
Disallow: /*?sort= # any URL containing ?sort=
Disallow: /*.pdf$ # URLs ending in .pdf
Disallow: /*/print/ # a print folder at any depth
An empty Disallow: means nothing is blocked. Disallow: / blocks everything.
Which rule wins
When several rules match a URL, Google uses the most specific one, measured by the length of the rule path. If an allow and a disallow are equally specific, Google uses the least restrictive, which is the allow. Order in the file is irrelevant.
| Rules | URL | Result | Why |
|---|---|---|---|
Disallow: /shop/ and Allow: /shop/sale/ |
/shop/sale/boots |
Allowed | The allow rule is longer |
Disallow: /shop and Allow: /shop |
/shop/item |
Allowed | Same length, least restrictive wins |
Disallow: /*.php and Allow: /index.php |
/index.php |
Allowed | /index.php is longer than /*.php |
Disallow: / and Allow: /$ |
/ |
Allowed | Only the homepage is allowed, everything else is blocked |
Not every crawler implements precedence the same way, so avoid clever overlapping rules where you can. Test what you write with the Robots.txt Tester, which evaluates a URL against a live file for a chosen user agent.
What Google does not support
Crawl-delay: ignored by Google. Bing and some others read it.Noindex:lines in robots.txt: never officially supported, and Google stopped honouring them entirely in 2019.Host:and other non-standard fields: ignored.
How Google handles fetch errors on robots.txt
Before crawling a host, Google fetches its robots.txt. What happens next depends on the response, and the server error case is the one that hurts sites.
| Response | Google's behaviour |
|---|---|
| 2xx | Parses the file and follows it |
| 3xx | Follows at least five redirect hops, then treats the file as a 404 |
| 4xx except 429 | Treated as if no robots.txt exists: no crawl restrictions |
| 429 and 5xx | Treated as a server error. Google stops crawling the site for the first 12 hours while retrying, then uses the last cached good copy for up to 30 days. After 30 days it assumes no restrictions if the rest of the site is reachable, or keeps treating the site as unavailable if it is not |
Two practical consequences:
- A missing robots.txt (404) is harmless.
- A robots.txt that returns 500 or 503, or times out, can pause crawling of the whole host. If your firewall or bot protection challenges Googlebot on
/robots.txt, you will see crawling drop across the site. The Crawl stats report in Search Console has a host status section that flags robots.txt fetch failures.
Google generally caches the file for up to 24 hours. After an urgent fix you can request a recrawl of the file from the robots.txt report (Settings, then robots.txt) in Search Console. That report also shows the last fetched version and any parsing warnings for each host in the property.
What to block and what to leave alone
Robots.txt is for URL spaces that have no search value and cost crawl resources.
Good candidates:
- Internal search result pages (
/search,/?s=). - Faceted navigation and sort parameters that create near infinite combinations (see faceted navigation and indexing).
- Cart, checkout and account flows.
- Calendar or "next month" links that go on forever.
- API endpoints that return data not needed for rendering pages.
Do not block:
- CSS, JavaScript and image files that pages need to render. Google renders pages like a browser, and blocked resources can leave it with a broken or empty layout. More in JavaScript SEO.
- Pages you want removed from the index. Blocking stops Google from seeing the noindex or the 404. This is the classic trap, explained in noindex vs robots.txt vs canonical.
- Duplicate URLs you want consolidated with a canonical tag. Google cannot read a canonical on a page it may not fetch.
- Old URLs that redirect. Google has to crawl them to see the redirect.
- Anything private. Robots.txt is public and only a request. Use authentication.
Examples you can adapt
Allow everything
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xml
WordPress
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://www.example.com/sitemap_index.xml
Do not block /wp-content/ or /wp-includes/. Themes and plugins load CSS and JavaScript from there.
Online shop with parameters
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /*?*sort=
Disallow: /*?*filter=
Disallow: /*?*sessionid=
Sitemap: https://www.example.com/sitemap.xml
The pattern /*?*sort= matches the parameter whether it is first in the query string or not.
Staging site
User-agent: *
Disallow: /
This keeps compliant crawlers out, but URLs can still be indexed from links. HTTP authentication is the reliable way to protect staging. And make sure this file never ships to production. A leftover Disallow: / after launch is the single most damaging robots.txt mistake, and it happens all the time.
Opting out of AI training crawlers
Several AI companies publish user agent tokens that can be disallowed separately from their search crawlers. For example, Google-Extended controls use of your content for Google's Gemini models and does not affect Google Search, and GPTBot is OpenAI's training crawler.
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
Check each vendor's documentation for current token names. The Robots.txt Generator includes the common ones.
Common mistakes
Blocking the site by accident. Disallow: / under User-agent: *, usually copied from staging. Confirm with a live test after every deployment.
A trailing slash changes the meaning. Disallow: /blog blocks /blog-news/ and /blogging-tips too. Disallow: /blog/ does not block /blog itself.
A specific group that drops the general rules. Covered above: Googlebot follows only its most specific group.
Expecting removal from search. Pages that are blocked but linked can show up as "Indexed, though blocked by robots.txt". The fix for that and for the opposite problem is in blocked by robots.txt: how to fix it.
Serving it dynamically and slowly. A robots.txt generated by the application on each request inherits every application outage. Serve it as a static file where you can.
Wrong content type or a BOM. Serve text/plain. Google tolerates a UTF-8 byte order mark, but other parsers may choke on the first line. An HTML error page served with status 200 at /robots.txt is parsed as a robots.txt with no valid rules, which means everything is allowed.
Blocking via robots.txt to "save crawl budget" on a small site. If you have a few thousand pages, Google can crawl all of them. Blocking things rarely helps, and it often hides problems. See crawl budget explained.
How to test changes safely
- Write the new rules in a copy of the file.
- List URLs that must stay crawlable (home, a category, a product, an article, a CSS file, a JS bundle) and URLs that must be blocked.
- Test every URL on both lists against the new rules.
- Deploy, then fetch the live file and confirm it is the version you expect, with status 200 and
text/plain. - In Search Console, open the robots.txt report and request a recrawl. Then run URL Inspection live tests on two or three important pages and check that "Crawl allowed?" says yes.
- Watch the Page indexing report over the following weeks for growth in "Blocked by robots.txt".
Robots.txt is a blunt tool that does one thing well. Keep the file short, comment each rule with the reason it exists, and treat every change like a code change with a test plan. Most sites need fewer than ten lines.
Frequently asked questions
- Does robots.txt stop a page from being indexed?
- No. It stops compliant crawlers from fetching the page. If other pages link to a blocked URL, Google can still index the bare URL without its content. To keep a page out of the index, allow crawling and use a noindex robots meta tag or X-Robots-Tag header, or protect the page with a login.
- What happens if my site has no robots.txt?
- Nothing bad. When robots.txt returns 404, Google treats it as if there are no crawl restrictions and crawls normally. A missing file is safer than a wrong one.
- How long does Google take to notice a robots.txt change?
- Google generally caches robots.txt for up to 24 hours, sometimes longer when it cannot refresh the file. You can ask for a faster refresh from the robots.txt report in Search Console.
- Does Google support crawl-delay in robots.txt?
- No. Google ignores crawl-delay. Some other crawlers, including Bing, read it. To slow Googlebot in an emergency, return 503 or 429 for a short period instead.
- Does the order of rules in robots.txt matter?
- Not for Google. The most specific matching rule wins, measured by the length of the rule path, and when an allow and a disallow rule are equally specific, the less restrictive one (allow) is used.