Faceted Navigation and Indexing: A Practical Guide

Faceted navigation lets visitors filter a list by colour, size, brand, price and more. When each combination has its own crawlable URL, a category of 500 products can expose hundreds of thousands of URLs. Google's documentation names faceted navigation as one of the most common sources of overcrawling: Googlebot spends its time on endless filter combinations, and your real pages are discovered and refreshed more slowly. The fix is to decide which few filter pages deserve indexing and to keep crawlers away from the rest.

How the problem grows

Take one category with four filters: 10 colours, 8 sizes, 15 brands, 5 price bands. Allowing one value per filter already gives thousands of combinations. Add multi-select, sort orders, items per page and pagination, and the total runs to millions for a single category.

/shoes
/shoes?color=red
/shoes?color=red&size=42
/shoes?size=42&color=red                  (same page, different parameter order)
/shoes?color=red&size=42&sort=price_asc
/shoes?color=red&size=42&sort=price_asc&view=48&page=3

Google explains why it cannot simply skip these: a crawler cannot know whether a URL is useful without fetching it first. So it fetches a lot of them. Google's documentation describes two results:

• Overcrawling. Server resources and crawl budget go to URLs with no search value.
• Slower discovery crawls. Time spent on facets is time not spent on new products and updated pages.

Signs that this is happening to you

Where	What to look for
Page indexing report, "Unsubmitted pages only" filter	Example URLs full of ?color=, &sort=, /filter/
"Crawled - currently not indexed", "Duplicate without user-selected canonical", "Alternate page with proper canonical tag"	Counts many times larger than your real page count
Settings > Crawl stats	High request volume, with sample URLs that are mostly parameters
Server logs	The share of Googlebot hits on parameter URLs. See log file analysis for Googlebot
New products	Days or weeks in "Discovered - currently not indexed" although they are in the sitemap

If you have 5,000 real pages and Search Console knows 400,000 URLs, facets are the first suspect.

Step 1: decide what deserves indexing

Sort every facet into one of three groups.

Group	Examples	Treatment
Index	Brand + category ("nike running shoes"), type + category, gender + category	Clean static URL, unique title and H1, self-canonical, in the sitemap, internally linked
Crawlable, not indexed	Rarely needed. Filters whose only job is to lead crawlers to products not reachable any other way	noindex, follow. It is better to fix product discovery with pagination and sitemaps
Never crawl	Multi-select combinations, size, price range, sort, view, in-stock toggle, session and tracking parameters	Block in robots.txt, or do not generate URLs for them at all

How to decide what goes in the first group:

1. Search demand. Do people search for the combination? Check your Search Console queries and keyword tools. "Red dresses" yes, "red dresses size 12 under 40" no.
2. Inventory. Enough products to make a useful page, and stable over time.
3. Distinctness. The page should show a meaningfully different set of products from its parent.

Most shops end up with one or two indexable facets and block the rest.

Step 2: give indexable facet pages proper URLs

Pages you want indexed should not look like filter states. Treat them as subcategories:

/shoes/nike/          instead of   /shoes?brand=nike
/shoes/trail/         instead of   /shoes?type=trail

Each one needs a unique <title> and H1 ("Nike Running Shoes"), a self-referencing canonical, an entry in the XML sitemap, crawlable links from the parent category, and ideally a short introduction. That keeps the rule simple: clean paths are indexable, query parameters are not.

Step 3: stop the rest being crawled

Google's faceted navigation documentation gives two main routes for URLs you do not need in search, plus two weaker signals.

Option A: robots.txt

The most direct method. Google's own example blocks the parameters with wildcards:

User-agent: Googlebot
Disallow: /*?*products=
Disallow: /*?*color=
Disallow: /*?*size=
Allow: /*?products=all$

A typical shop version:

User-agent: *
Disallow: /*?*color=
Disallow: /*?*size=
Disallow: /*?*price=
Disallow: /*?*sort=
Disallow: /*?*view=

Notes:

• /*?*color= matches color= anywhere in the query string, whether it follows ? or &.
• Be careful with short names. Disallow: /*?*s= also matches ?colors=. Test real URLs of each kind (category, facet, product, paginated page) in the Robots.txt Tester before and after deploying.
• Do not block page= if pagination is how Google reaches deep products.
• Blocked URLs that have links can still show up as "Indexed, though blocked by robots.txt". For facet URLs this is uncommon and mostly harmless.

More syntax in the robots.txt guide.

Option B: URL fragments

Apply filters on the client and keep the state after a #:

/shoes#color=red&size=42

Google's documentation notes that Google Search generally does not support URL fragments in crawling and indexing, so these filter states create no crawlable URLs at all. The cost is that no filtered view can be indexed, which is why you keep the static URLs from step 2 for the ones that matter.

A related approach is to apply filters with JavaScript and no URL change, or through POST requests. It works for crawl control and is worse for users, who cannot share or bookmark a filtered view. Fragments avoid that problem.

Weaker signals: canonical and nofollow

Google lists both as useful and as less effective over the long term than the options above:

• rel="canonical" from filtered URLs to the unfiltered category. It consolidates signals and may reduce crawling of the variants over time. It is a hint. Google still has to crawl each variant to see the tag, and it can ignore the canonical when the filtered page's content differs a lot from the target.
• rel="nofollow" on links to filtered URLs. Google points out that every link to a given URL needs the attribute for this to work, internal and external, which is hard to guarantee.

Use canonical tags as a safety net on every facet URL. Do not make them your only defence on a large catalogue. Verify the output with the Canonical Tag Checker.

What about noindex?

noindex keeps facet pages out of the index, and Google tends to crawl noindexed pages less often as time goes on. Each URL still has to be fetched at least once, and then periodically, so it does not solve a crawl volume problem. It is suitable for small sites where the concern is index bloat and not crawl load.

Never combine noindex with a robots.txt block on the same URLs. Google cannot read a tag on a page it may not fetch. The three mechanisms are compared in noindex vs robots.txt vs canonical.

If you do want parameter-based facets crawled

Google's documentation gives three best practices for that case:

1. Use the standard & separator between parameters. Commas, semicolons and brackets are hard for crawlers to recognise as separators.
2. If filters are encoded in the path, keep their logical order the same every time and do not allow duplicate filters. /shoes/red/42/ and /shoes/42/red/ must not both exist.
3. Return a 404 when a filter combination has no results. Not a 200 page saying "0 products", and not a redirect to a general error page. The same applies to nonsensical or duplicate combinations.

A few more from practice:

• Normalise parameter order on the server. If ?size=42&color=red is requested, 301 it to ?color=red&size=42, or always generate links in one fixed order.
• Drop empty and default parameters. ?color=&sort=default should not exist.
• Keep tracking parameters out of internal links.
• Do not list facet URLs in the XML sitemap unless they belong to the indexable group.

Make sure products stay discoverable

Blocking facets removes crawl paths. Before you deploy, confirm that every product is reachable without them:

• Categories have crawlable pagination with plain <a href> links.
• Every product is in the XML sitemap with an accurate lastmod.
• Products are linked from at least one category page that is not blocked.
• Related product modules link products to each other.

A crawl of your own site with the facet parameters excluded should still find every product. If it does not, you have created orphan pages.

Rollout and what to expect

1. Inventory the parameters. Export URLs from logs or a crawl and list every parameter name with its URL count. The URL Parser helps to break down unfamiliar URLs.
2. Classify each as index, crawl-only or block.
3. Build the static URLs for the indexable facets, and 301 the old parameter versions to them.
4. Add canonicals to all remaining facet URLs.
5. Add robots.txt rules and test them against sample URLs of every type.
6. Deploy and watch Crawl stats for two to four weeks.

After deployment:

• Googlebot requests to parameter URLs drop within days, as soon as the new robots.txt is fetched.
• Already indexed facet URLs are not removed by robots.txt. Most were never indexed. If a meaningful number are, let Google recrawl them with noindex or a 404 first, wait until they drop out, and add the robots.txt block afterwards.
• "Blocked by robots.txt" in the Page indexing report will grow. That is the intended result.
• Discovery and refresh crawling of real pages should improve over the following weeks. Check the "By purpose" breakdown in Crawl stats, and how long new products take to be indexed.

Common mistakes

• Blocking with robots.txt and adding noindex together, then wondering why the URLs stay indexed.
• Overly broad patterns. Disallow: /*? blocks every parameter, including pagination, and sometimes versioned CSS and JavaScript files that Google needs for rendering.
• Canonicalising everything to the category and calling it done on a site with millions of combinations. The crawling continues.
• Indexing every facet for long-tail traffic. Thousands of near-identical pages with three products each are thin content, and they drag down how Google sees the site.
• Linking to blocked facets from the main navigation, which sends mixed signals. Navigation should use the clean indexable URLs.
• Expecting the URL Parameters tool to handle it. Google removed that tool in 2022.
• Forgetting internal search. /search?q= creates the same infinite space. Disallow it.

Facet control is one of the largest crawl budget levers on an ecommerce site. The wider model is in crawl budget explained.