Crawl Budget Explained: Who Needs to Care and What to Do

Crawl budget is the number of URLs Googlebot can and wants to crawl on your site in a given period. It is set by two things: the crawl capacity limit (how much your server can take) and crawl demand (how much Google wants your content). For most sites it is not a constraint at all. It becomes a real issue on very large sites, on sites that generate huge numbers of low value URLs, and on servers that are slow or unstable.

Who should care

Google's own guide for large site owners says it is aimed at:

• Large sites (around a million unique pages or more) whose content changes moderately often, about once a week.
• Medium or larger sites (around 10,000 pages or more) whose content changes very rapidly, daily.
• Sites with a large share of URLs classified as "Discovered - currently not indexed" in Search Console.

Google notes these numbers are rough estimates. If you run a 300 page company site or a blog with 2,000 posts and your new pages are slow to get indexed, the cause is almost certainly not crawl budget. Look at quality, internal links and duplication first. Why is my page not indexed covers those causes.

The third bullet is the interesting one. A small site can still create a crawl budget problem for itself by exposing hundreds of thousands of parameter URLs. A shop with 800 products and a faceted navigation that allows any combination of six filters has, as far as a crawler is concerned, millions of pages.

The two components

Crawl capacity limit

Googlebot tries not to overload your server. It works out a limit on parallel connections and the delay between fetches, and adjusts it continuously:

• Crawl health. If the site responds quickly for a while, the limit goes up. If responses slow down or the server returns 5xx errors or 429s, the limit goes down.
• Google's own resources. Google has a lot of machines, but not infinitely many, and it allocates them.

The old crawl rate limiter setting in Search Console was retired in early 2024. Today, the ways a site owner can lower the rate are returning 500, 503 or 429 responses for a short period, or filing a request through the Googlebot report form. There is no way to raise it on request.

Crawl demand

Even with capacity to spare, Google only crawls as much as it wants to. Demand depends on:

• Perceived inventory. Without guidance, Googlebot tries to crawl all the URLs it knows about on your site. If many of them are duplicates, removed or unimportant, that is wasted time. This is the factor you control most directly.
• Popularity. URLs that are more popular on the web are crawled more often to keep them fresh.
• Staleness. Google wants to recrawl often enough to pick up changes.

Site wide events such as a domain move also trigger a burst of demand, because everything has to be reprocessed at the new URLs.

Put together: crawl budget is the set of URLs Google can and wants to crawl. If demand is low, Google crawls less even when the capacity limit is nowhere near reached. This is why speeding up a server does not help a site whose pages Google does not find valuable, and why improving content does not help a site that answers every third request with a 503.

How to tell if you have a problem

The Crawl stats report

In Search Console, go to Settings, then Crawl stats. It shows the last 90 days.

Metric	What to look for
Total crawl requests	The trend. A sudden drop without a site change points to server trouble or a robots.txt issue.
Average response time	Steady and low is good. Rising response time is followed by falling crawl requests.
Host status	Any warnings about robots.txt fetch, DNS resolution or server connectivity.
By response	The share of 200 versus 301, 404 and 5xx. A large 404 or redirect share is wasted crawling.
By purpose	Discovery (new URLs) versus Refresh (known URLs). If you publish a lot and discovery is a tiny share, new URLs are not being reached.
By file type	A very high share of JSON or script requests can mean rendering is expensive.

Click into any row to see sample URLs. This is often where you find the waste: thousands of fetches to ?sort=, ?sessionid= or calendar pages.

A rough ratio

Take the number of indexable pages on your site and divide by the average pages crawled per day from the report. If the result is a large number of days and your content changes faster than that, crawling is a bottleneck. If the number is small, crawling is fine and your problem is elsewhere. Google suggests a similar comparison in its documentation.

Server logs

Crawl stats is sampled. Logs show every request, so log file analysis is how you see which sections Googlebot spends its time on.

The indexing report

A large, growing "Discovered - currently not indexed" bucket on a big site is the classic symptom: Google knows the URLs and has not got round to fetching them. See discovered, currently not indexed. Be careful with the diagnosis, though. The same status on a small site usually reflects Google's doubt about quality, not a capacity limit.

What actually helps

1. Reduce the inventory of worthless URLs

This is where most gains are.

• Block infinite spaces in robots.txt. Faceted filters, sort orders, internal search, calendars and session parameters.

User-agent: *
Disallow: /*?*sort=
Disallow: /*?*price_min=
Disallow: /search
Disallow: /calendar/

Google's guidance is direct on this: use robots.txt for pages you do not want crawled, and do not use noindex for it, because Google still has to request the page to see the tag. Note the trade-off: a blocked URL cannot pass a canonical or a noindex. Decide per URL pattern which you need. The options are compared in noindex vs robots.txt vs canonical, and the filter case in faceted navigation and indexing.

• Consolidate duplicates. One URL per piece of content. Fix internal links that point to parameter or non-canonical versions, because every such link is an invitation to crawl.
• Return 404 or 410 for pages that are gone. Google keeps retrying URLs it knows. A clear 404 tells it to stop sooner than a blocked URL or a soft 404 does.
• Eliminate soft 404s. Pages that say "not found" or "no results" with a 200 status get crawled again and again. See soft 404 errors.

2. Remove friction

• Avoid long redirect chains. Each hop is a separate fetch. Link directly to the final URL.
• Make the server fast and stable. Response time and error rate feed straight into the capacity limit. Cache HTML where you can, and make sure bot protection is not throttling Googlebot.
• Support conditional requests. If your server answers If-Modified-Since or If-None-Match with 304 Not Modified when nothing changed, Google saves the download and you save the rendering work.

GET /guide HTTP/1.1
If-Modified-Since: Tue, 01 Sep 2026 08:00:00 GMT

HTTP/1.1 304 Not Modified

• Keep rendering cheap. Pages that need dozens of script and API requests to render cost more to process. Embedded resources count against the same budget.

3. Point Google at what matters

• Keep sitemaps current, with truthful lastmod. Google reads sitemaps regularly, and an accurate <lastmod> helps it pick changed pages. See lastmod best practices.
• Strengthen internal links to new and important pages. Pages linked from the homepage or main hubs are found and recrawled faster than pages six clicks deep.

Myths worth dropping

Claim	Reality
"Every site should optimise crawl budget"	Google states that most sites do not need to think about it.
"Crawl-delay in robots.txt controls Googlebot"	Googlebot ignores crawl-delay.
"Noindex frees up crawl budget"	The page is still fetched. It may be crawled less often over time, but it is not a crawl control.
"Temporarily blocking sections shifts budget to other sections"	Google advises against this. It will not move the freed capacity elsewhere unless your site is already hitting its serving limit.
"More crawling means better rankings"	Crawling is not a ranking signal.
"Nofollow on internal links saves budget"	A nofollowed link may keep a URL from being discovered through that link, but any other link or sitemap entry still exposes it.

If Googlebot crawls too much

The opposite problem exists too. First verify that the traffic really is Googlebot, then look for a newly exposed infinite URL space, which is the usual cause. For an emergency, Google documents returning 503 or 429 for a short period, never more than a day or two. The procedure is in server errors and Googlebot.

A sensible order of work

1. Confirm you are in one of the groups that should care.
2. Open Crawl stats, and look at response codes and sample URLs for waste.
3. Fix server errors and response time first. Nothing else matters while the capacity limit is being pushed down.
4. Cut worthless URL inventory with robots.txt, 404s and canonical clean-up.
5. Make sitemaps accurate and internal links direct.
6. Measure again after a few weeks. Crawl behaviour changes gradually.

For everyone else, the practical takeaway is a relief: publish good pages, link to them, keep a clean sitemap, and let Googlebot get on with it. You can spot check how crawlable a key URL is with the Indexability Checker.