Skip to content
TurboIndex
Sign in

Robots.txt Tester

Fetch the live robots.txt of any site, pick a crawler and test up to 20 paths. You see the verdict and the exact rule behind it.

How the test works

The tool downloads /robots.txt from the host you enter, picks the group of rules that applies to the chosen user agent and matches every path against it, following the rules Google documents for its own parser:

  • A crawler uses the most specific User-agent group that matches its name. Googlebot-Image prefers a googlebot-image group, then googlebot, then *.
  • Within the group, the longest matching rule wins. On a tie, Allow beats Disallow.
  • * matches any run of characters and $ anchors the end of the URL. Disallow: /*.pdf$ blocks /file.pdf but not /file.pdf?v=2.
  • Paths are case sensitive. /Admin/ and /admin/ are different.
  • When no rule matches, the path is allowed.

You can paste full URLs. Only the path and query string are tested, because robots.txt rules never include the host.

AI crawlers

GPTBot (OpenAI) and ClaudeBot (Anthropic) respect robots.txt. Blocking them does not affect Google rankings, since they are separate from Googlebot. Pick them in the user agent list to confirm your rules do what you intended. For other bots choose Custom and type the token, for example AhrefsBot or CCBot.

Mistakes this tool catches

A leftover Disallow: /

Staging sites often block everything. If that file ships to production, every path shows as blocked here.

Blocked CSS and JavaScript

Google renders pages. Test a few asset paths such as /wp-includes/ or /assets/app.js. If they are blocked, Google may see a broken layout.

Blocking a page you want removed

Google cannot see a noindex tag on a page it may not fetch. Use the indexability checker to see robots.txt, noindex and canonical signals for one URL together.

Next steps

Need a new file? The robots.txt generator builds one. If the file declares no sitemap, locate yours with the sitemap finder and add a Sitemap: line. When crawling is open and pages are still missing from Google, TurboIndex submits them for you and tracks the result.

Questions and answers

Which rule wins when Allow and Disallow both match?

The most specific rule, which means the one with the longest path. If an Allow and a Disallow rule are equally long, Google uses the less restrictive one, so Allow wins. The order of the lines in the file does not matter.

Does Googlebot follow the rules under User-agent: *?

Only when the file has no group that names Googlebot. A crawler obeys the single most specific group that matches its name and ignores the rest. If you add a Googlebot group, repeat in it every rule from the * group that should still apply.

Does Disallow remove a page from Google?

No. Disallow stops crawling, not indexing. A blocked URL that other sites link to can still appear in results without a description. To keep a page out of the index, allow crawling and use a noindex tag.

What happens when robots.txt is missing or returns an error?

A 404 or other 4xx answer means there are no restrictions and everything may be crawled. A 5xx error is different: Google stops crawling the site for a while, and if the error lasts it falls back on the last copy of the file it has cached.

Is there a size limit for robots.txt?

Google reads the first 500 KiB of the file and ignores the rest. Rules after that point have no effect, so keep the file small by using wildcards instead of listing thousands of paths.