Indexed, Though Blocked by robots.txt: What to Do

"Indexed, though blocked by robots.txt" means Google added a URL to its index without fetching the page, because a Disallow rule in robots.txt stops Googlebot from crawling it while links from other pages tell Google it exists. The index entry contains no content from the page itself. The fix depends on what you want: if the page should be in Google, remove the block so it can be crawled; if it should not be, remove the block and add noindex, because a page Google cannot fetch is a page whose noindex Google cannot read.

What the state means

Google's index has entries for pages it has crawled and for URLs it only knows by reference. When robots.txt blocks a URL, Google is not allowed to request it, but it still sees links pointing at it: from your own navigation, from your sitemap, from other sites. If those references look important enough, Google indexes the URL with what it has: the URL string and the anchor text of the links.

In search results, such a page looks broken. The title is the URL or an anchor text, and where the description would be, Google says that no information is available for the page. It can still rank for its own URL or for a query that matches the anchor text, which is how private admin paths and staging hosts occasionally show up in results.

The state is different from plain "Blocked by robots.txt", which is listed among not indexed pages: same block, but Google chose not to index the URL. Which of the two a blocked URL lands in depends on how many links point at it and how important Google judges it. The repair for both is in Blocked by robots.txt: how to fix it; this article is about the indexed variant and the two opposite fixes it calls for.

Why it happens

Almost always one of these:

• A rule meant for crawl control hits pages you link to. Disallow: /products/ to stop parameter crawling also blocks the real product pages, which every category page links to.
• Pages others link to. A PDF, a tool, an old URL that picked up links before you blocked it.
• Staging or development hosts protected only with Disallow: /. Someone shared a link, and the URL went in.
• Filter and sort parameters that the site generates links to (?sort=price, ?color=red) and robots.txt blocks. Thousands of these is the typical shape of this warning on shops.
• Blocking after indexing. The pages were crawled and indexed, then a block was added. Google keeps the entries, now without fresh content, and reports them here.
• Platform defaults. Some CMS templates block directories such as /wp-content/uploads/ or /media/ while linking to files in them.

Where to find the affected URLs

Open Indexing > Pages in Search Console. Scroll past the chart to the tables; the state is listed with the indexed pages that carry a warning (the table is titled along the lines of "Improve page appearance"). Click the row to see up to 1,000 example URLs, and export them if there are more.

Then inspect two or three with URL Inspection. The Page indexing section reads:

• Crawl allowed? No: blocked by robots.txt
• Page fetch Failed: Blocked by robots.txt
• Indexing allowed? N/A or unknown, because Google never saw the page

The same inspection shows Referring page where it is known, which tells you which link made Google care about the URL.

To see the rule that matches, put the URL into the robots.txt tester with Googlebot as the agent. It reports the exact line. Remember that robots.txt is per host: check the file on the host of the affected URL, not on the main site, if they differ.

Decide what you want for each group

Sort the exported URLs into two piles. The pattern usually sorts itself by directory or parameter.

You want	The fix	Why
The page in Google, properly, with its content	Remove or narrow the Disallow, let Google crawl it	Google needs the content to rank it and to show a real snippet
The page out of Google	Remove the Disallow for those URLs, add noindex, let Google crawl it	Google only honours noindex on pages it can fetch
Nothing: it is a junk URL nobody searches for	Either of the above, or leave it	The warning is cosmetic for URLs with no search value

The one thing that does not work is adding noindex and keeping the block. The tag sits unread and the URL stays indexed.

Fix A: you want the page indexed

1. Edit robots.txt. Narrow the rule so it stops matching the pages you want crawled. If Disallow: /products/ was meant to stop parameters, replace it with Disallow: /products/*? or Disallow: /*?sort=.
2. Test the changed file against three affected URLs and three URLs that should still be blocked, so the fix does not open the door too wide.
3. Deploy, then open Settings > Crawling > robots.txt > Open report in Search Console to confirm Google has fetched the new version. Google generally refetches robots.txt within 24 hours; the report has an option to request a recrawl sooner.
4. Inspect the most important affected URL and click Request indexing. For the rest, open the warning's row in the Page indexing report and click Validate fix.

After the next crawl, the entry gets real content: a proper title, a description and the page's own text in the index. The warning clears for that URL as the report catches up.

Fix B: you want the page out of Google

The order matters here because Google has to crawl the page in order to see that it should not be indexed.

1. Remove the Disallow for the URLs in question, or add an Allow line that overrides it for them.
2. Add noindex: a <meta name="robots" content="noindex"> tag in the HTML head, or an X-Robots-Tag: noindex response header for files such as PDFs and images. Server examples for both are in the noindex guide.
3. Deploy and confirm with the indexability checker that the URL now returns 200 with the noindex directive and is allowed by robots.txt.
4. Request indexing for the handful you care about most (yes, request indexing, so that Google fetches the page and reads the tag), and use Validate fix for the group.
5. Once the URLs show as "Excluded by 'noindex' tag" in the report, you can put the robots.txt block back if you need it for crawl capacity. Most sites do not need to; a noindex page costs little to crawl occasionally.

Alternatives to noindex for step 2, depending on the page:

• A page that no longer exists: return 404 or 410. Google drops it after the next crawl.
• Private content: put it behind authentication so Google receives 401. Google does not index 401 responses and treats the page as gone.
• A parameter variant of a real page: a rel="canonical" to the real page, once crawling is allowed. This consolidates rather than removes, which is usually the better outcome for shop filters.

Which of these fits which situation is laid out in noindex vs robots.txt vs canonical.

Urgent cases: the Removals tool

If the URL exposes something that should not be public, do not wait for the recrawl. In Search Console open Indexing > Removals, click New request, enter the URL (or a prefix to cover a whole directory) and submit. Google hides it from results, usually within a day, for about six months. It is a temporary measure: the permanent fix is still one of the options above, applied during those six months, otherwise the URL comes back when the removal expires.

The Removals tool works on a URL that is blocked in robots.txt, which makes it the one tool that acts immediately in this situation. It does not remove the index entry; it suppresses the result.

How long it takes

• robots.txt refetch: up to about a day, sooner with a recrawl request in the robots.txt report.
• Recrawl of the affected URLs: days for pages Google visits often, weeks for deep or low-priority ones.
• Page indexing report: lags the crawl by days. Do not judge the fix by the report alone; inspect a few URLs directly.
• Validate fix: Google's documentation says validation typically takes up to about two weeks and can take longer.

When the warning matters and when it does not

For a handful of parameter URLs with no search value, no. The warning is not an error, it does not affect the rest of the site, and the entries are harmless. It is worth fixing when:

• the affected URLs are pages you want traffic on (then the index has nothing to rank),
• the affected URLs expose paths you do not want visible (staging, admin, internal documents),
• the count is large and growing, which usually means a rule is wrong and is also costing you crawl capacity on the pages that are blocked and not indexed.

A large count of blocked but indexed shop filters is also a signal to look at the site's URL generation rather than at robots.txt: fewer linked parameter variants means fewer blocked URLs to worry about, with or without the rule. Faceted navigation and indexing covers that side.

Keeping blocked URLs out of your submissions

Submitting a blocked URL for indexing is a wasted request: Google cannot fetch it, and the submission just reinforces that the URL exists. TurboIndexBot, the crawler TurboIndex runs over each sitemap URL before submission, fetches the page first, so a URL that returns an error, redirects or carries a noindex is held back with that reason. Keep in mind that it obeys rules addressed to it by name rather than the * group, so a URL that Googlebot is blocked from can still pass its check; the robots.txt tester as Googlebot remains the authoritative test for Google. What the crawler requests and how to control it is on the TurboIndexBot page.

What to do next

1. Export the affected URLs and group them by directory or parameter.
2. For each group, decide: indexed properly, out of Google, or ignore.
3. Edit robots.txt so Google can crawl every group that is not in the ignore pile, and add noindex to the groups that should be out.
4. Confirm the new robots.txt in the Search Console report, request indexing for the top URLs, and validate the fix.
5. Check the warning count again in two to four weeks.