Blocked by robots.txt in Search Console: How to Fix It

"Blocked by robots.txt" means Googlebot wanted to crawl a URL and a disallow rule in your robots.txt told it not to. Whether that is a problem depends entirely on the URL. For a checkout page it is the intended result. For a product or article page it means Google cannot read the content, and the fix is to find the rule that matches and change it. The related status "Indexed, though blocked by robots.txt" needs a different fix, covered below.

Two statuses, two different problems

Status in Page indexing report	What happened	Is the URL in Google?
Blocked by robots.txt	Google knows the URL, was not allowed to fetch it, and has not indexed it	No (at the time of the report)
Indexed, though blocked by robots.txt	Google was not allowed to fetch it, and indexed the URL anyway based on links pointing to it	Yes, typically with no snippet

The second one exists because robots.txt is a crawling control. It says "do not fetch", not "do not list". If enough links point at a blocked URL, Google may decide the URL is worth showing and build a result from the anchor text alone.

Step 1: decide whether each URL should be blocked

Open Search Console, go to Indexing, then Pages, and click the "Blocked by robots.txt" row. Export the list. Sort the URLs into two piles:

• Blocked on purpose: admin, cart, checkout, account, internal search, filter and sort parameters, API endpoints. Leave these alone. The report is informational.
• Should be in Google: any page with content you want to rank. These need fixing.

If pile two is empty, you are done. Many site owners spend time "fixing" this report when there is nothing wrong.

One refinement: switch the report filter at the top from "All known pages" to "All submitted pages". Any URL that appears here is both in your sitemap and blocked, which is a contradiction you should resolve one way or the other. Either unblock it or take it out of the sitemap.

Step 2: find the rule that matches

Run URL Inspection on one affected URL. Under Page indexing, "Crawl allowed?" will say "No: blocked by robots.txt". Then find the rule.

Open https://yourdomain.com/robots.txt in a browser and check these in order:

1. The right host. Robots.txt is per host and protocol. https://www.example.com and https://example.com have separate files. Look at the one for the host in the blocked URL.
2. The right group. Googlebot follows only the most specific matching User-agent group. If there is a User-agent: Googlebot group, the rules under User-agent: * do not apply to it at all.
3. Prefix matches. Disallow: /p blocks /products/, /pricing and /press. Disallow: /blog blocks /blog-archive/ as well.
4. Wildcards. Disallow: /*? blocks every URL with a query string. Disallow: /*.html blocks every URL containing .html anywhere.
5. Case. Paths are case sensitive.

The fastest route is to paste the URL into the Robots.txt Tester, choose Googlebot, and let it report which line matched.

Typical culprits:

# Left over from staging
User-agent: *
Disallow: /

# Meant to block one folder, blocks half the site
Disallow: /p

# Meant to block tracking parameters, blocks all paginated content
Disallow: /*?

# WordPress "Discourage search engines" setting on older versions
User-agent: *
Disallow: /

On WordPress, check Settings, then Reading, and make sure "Discourage search engines from indexing this site" is unticked. On current versions that setting adds a noindex tag instead of a robots.txt block, but plugins and older installs can still write a disallow rule to the virtual robots.txt. On Shopify, the default file blocks certain paths by design (cart, checkout, some filtered collection URLs) and you edit it through the robots.txt.liquid theme template. There is more platform detail in WordPress indexing problems and Shopify indexing problems.

If you cannot find a matching rule in the live file, Google may be working from an older cached copy, or the file may differ for Googlebot. Open the robots.txt report in Search Console (Settings, then robots.txt) to see the exact version Google last fetched and when.

Step 3: fix the rule

Choose the narrowest change that solves it.

Remove the rule if it has no purpose:

 User-agent: *
-Disallow: /p
 Disallow: /cart/

Narrow the rule if it caught too much:

-Disallow: /blog
+Disallow: /blog/drafts/

Add an allow exception if most of a folder should stay blocked:

User-agent: *
Disallow: /resources/
Allow: /resources/guides/

The longer, more specific rule wins for Google, so /resources/guides/seo is crawlable while the rest of /resources/ stays blocked.

Replace a blanket parameter block with specific ones:

-Disallow: /*?
+Disallow: /*?*sort=
+Disallow: /*?*sessionid=

After editing, test both directions: the pages you wanted open are allowed, and the pages you wanted closed are still blocked.

Step 4: get Google to notice

1. Confirm the live file shows your change and returns status 200.
2. In Search Console, go to Settings, then robots.txt, and use "Request a recrawl" on the file. Without this, Google generally picks up the change within about 24 hours, since that is how long it normally caches the file.
3. Run a URL Inspection live test on a previously blocked URL. "Crawl allowed?" should now say yes. For your most important pages, click Request indexing. That feature has a small daily quota, described in request indexing limits.
4. Back in the Page indexing report, open the "Blocked by robots.txt" issue and click Validate fix. Validation can take a couple of weeks because Google has to recrawl the sample.
5. Make sure the unblocked URLs are in your sitemap, so they are rediscovered quickly.

Unblocking makes the pages crawlable. It does not guarantee they get indexed. After Google reads them, they go through the same quality evaluation as any other page.

Fixing "Indexed, though blocked by robots.txt"

Here the question is reversed: do you want this URL in Google?

Yes, I want it indexed properly. Remove the block, as above. Once Google can crawl the page it will show a real title and snippet.

No, I want it gone. This is where people go wrong. Adding a noindex tag while the block stays in place does nothing, because Google never fetches the page and never sees the tag. The correct sequence:

1. Remove the disallow rule for those URLs.
2. Add noindex, either in the HTML or as a header:

<meta name="robots" content="noindex">

X-Robots-Tag: noindex

3. Wait until Google has recrawled the URLs and dropped them. Check with URL Inspection.
4. Only then, if crawl volume is a concern, put the disallow back. For most sites it is fine to leave the pages crawlable with noindex permanently.

If the content is sensitive and needs to disappear quickly, use the Removals tool in Search Console for a temporary hide (about six months) while the noindex takes effect, and put real access control on the page. The full decision logic between the three mechanisms is in noindex vs robots.txt vs canonical.

It does not matter. Blocked parameter URLs sometimes get indexed without content and never receive impressions. If they are not showing for real queries, you can ignore them. Google's own help text says this status is often not a problem when the block is intentional.

Blocked resources are a separate issue

The Page indexing report covers pages. Googlebot can also be blocked from CSS, JavaScript, fonts or API calls that your pages need in order to render. That does not show in this report. You find it in URL Inspection: run a live test, open View tested page, then More info, then Page resources, and look for entries marked as blocked by robots.txt. If a blocked script is what loads your main content, Google indexes an empty shell. Unblock anything required for rendering:

User-agent: *
Disallow: /assets/private/
Allow: /assets/*.css
Allow: /assets/*.js

Preventing a repeat

• Keep robots.txt in version control and review changes like code.
• Make the deployment pipeline fail if the production file contains Disallow: / on its own line.
• Keep staging protected by HTTP authentication, so it never needs a blocking robots.txt that could leak into production.
• After each release, check a handful of key URLs with the Indexability Checker, which tests robots.txt, status code, noindex and canonical in one pass.
• Monitor the "Blocked by robots.txt" count in the Page indexing report. A sudden jump right after a release is almost always a rule change.

For the full syntax, precedence rules and how Google treats robots.txt errors, see the robots.txt guide.