Server Errors (5xx) and Googlebot: Causes, Risks and Fixes

A 5xx status code tells Googlebot that your server failed to handle the request. Google's response has two parts: it slows its crawling of your site, and if the errors continue it eventually removes the affected URLs from the index. A short outage is harmless, because already indexed pages are kept while Googlebot retries. Days of persistent errors are not. So the priority is always the same: find out what Googlebot is hitting and restore a clean 200.

How Google reacts, step by step

According to Google's documentation on HTTP status codes:

• 5xx and 429 responses prompt its crawlers to temporarily slow down.
• The reduction in crawl rate is proportionate to the number of URLs returning errors.
• Already indexed URLs are preserved in the index, but eventually dropped if the errors persist.
• Once the server responds normally again, the crawl rate recovers gradually.

Network level failures (timeouts, connection resets, DNS errors) are handled much like 5xx.

There is one case with a larger blast radius: robots.txt. Googlebot fetches /robots.txt before crawling. If that file returns a 5xx, Google stops crawling the site for the first 12 hours while it retries, then falls back to the last good cached copy for up to 30 days. After that it assumes there are no restrictions if the rest of the site is reachable. A broken robots.txt can therefore pause crawling of an entire host even when all the pages are fine. Details in the robots.txt guide.

The codes you will see

Code	Meaning	Usual culprit
500 Internal Server Error	The application crashed or threw an unhandled exception	Code bugs, bad input handling, failed database queries, PHP fatal errors, broken .htaccess
502 Bad Gateway	The proxy or CDN got an invalid response from the origin	App server (PHP-FPM, Node, Gunicorn) crashed or restarting, origin refused the connection
503 Service Unavailable	The server is temporarily unable to respond	Maintenance mode, overload, exhausted worker or connection pool, rate limiting
504 Gateway Timeout	The proxy or CDN waited too long for the origin	Slow database queries, heavy uncached pages, origin under load
520 to 527	Cloudflare-specific origin errors	Origin unreachable, TLS handshake failures, origin timeouts
429 Too Many Requests	Not a 5xx, but Google treats it as a server error	Rate limiter or bot protection throttling Googlebot

Where to see the problem

Page indexing report

Indexing, then Pages, then the row "Server error (5xx)". This lists URLs where Google's last crawl attempt failed. Export it and look for a pattern: one template, one folder, only URLs with parameters, only very old URLs.

Crawl stats report

Settings, then Crawl stats. Three things to look at:

• Host status. It reports problems in three categories: robots.txt fetch, DNS resolution and server connectivity. A red or amber state here means the failures were serious enough to affect crawling of the whole host.
• By response. The share of "Server error (5xx)" among all requests, with example URLs and timestamps.
• The time series. A falling line for total crawl requests together with a rising line for average response time is the signature of an overloaded server: Googlebot sees things slowing down and backs off.

URL Inspection

Run a live test on an affected URL. If it succeeds, the problem is intermittent or already fixed. If it fails, you can reproduce it on demand, which makes debugging much easier.

Your logs

Search Console samples. Your access logs have everything. Count Googlebot requests by status code per day:

grep "Googlebot" access.log \
  | awk '{print substr($4,2,11), $9}' \
  | sort | uniq -c

Then list the URLs that returned 5xx to Googlebot:

grep "Googlebot" access.log | awk '$9 ~ /^5/ {print $9, $7}' \
  | sort | uniq -c | sort -rn | head -30

If errors happen at the CDN or load balancer, the origin's access log may show nothing at all. Check the logs at the layer that produced the error. For field positions and for verifying that the requests really come from Google, see log file analysis for Googlebot.

Why Googlebot sees errors you do not

It requests URLs nobody else does. Old URLs, parameter combinations, malformed paths. A route that throws an exception on unexpected input returns 500 where it should return 404 or 400. A typical example: /products?page=99999 or /products?page=abc causes an unhandled error in the pagination code. Validate input and return 404 for out of range pages.

It arrives in bursts. A few requests per second, spread across many uncached pages, is a different load from regular users who mostly hit cached popular pages. Long tail pages that each need an expensive database query are what tip a small server over.

Your protection layer treats it as an attack. WAF rules, rate limiters and bot management products sometimes challenge or block Googlebot, and the result can be 403, 429 or 503. Most vendors keep an allowlist of verified search engine bots. Make sure it is switched on, and do not allowlist by user agent alone, since anyone can fake that.

Timeouts at the proxy. The origin would eventually answer in 40 seconds, but the CDN gives up at 30 and sends a 504. Users never request that heavy page. Googlebot does.

Deployments. Each release causes a minute of 502s while app servers restart. With several releases a day, Googlebot gets them regularly. Use rolling restarts or health-checked load balancing so that no request hits a server that is down.

Exhausted pools. PHP-FPM pm.max_children reached, database max_connections reached, Node event loop blocked. The server log usually says so explicitly:

WARNING: [pool www] server reached pm.max_children setting (20), consider raising it

Fixing it

1. Reproduce. Request an affected URL several times, with a Googlebot user agent, to catch cases where behaviour differs by user agent:

curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" \
  -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  https://www.example.com/affected-url

You will not be coming from a Google IP, so rules based on IP will not trigger, but rules based on user agent will. For checking many URLs at once there is the Bulk HTTP Status Checker.

2. Read the error log at the time of the failure. The application error log, not the access log, tells you why. Match timestamps from the Crawl stats examples.

3. Fix the class of problem, not one URL. Input validation, a missing database index, a larger worker pool, a caching layer for anonymous traffic, a firewall exception for verified Googlebot.

4. Make error handling honest. Missing things should return 404. Bad input should return 400 or 404. Only real server failures should be 5xx. And an error should never be served with status 200, because that creates soft 404s or indexes your error message.

5. Cache what you can. Full page caching for anonymous requests removes most of the load Googlebot generates.

6. Validate in Search Console. In the Page indexing report, open "Server error (5xx)" and click Validate fix. Google rechecks the affected URLs over the following days or weeks.

Planned downtime

If you know the site will be unavailable, return 503 with a Retry-After header for all URLs except robots.txt, which should keep returning 200 (or 404 if you have none).

Apache:

RewriteEngine On
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteCond %{REQUEST_URI} !^/maintenance\.html$
RewriteRule ^ - [R=503,L]
ErrorDocument 503 /maintenance.html
Header always set Retry-After "3600" "expr=%{REQUEST_STATUS} == 503"

Keep the window short. Hours are fine. Google's guidance on using error codes to slow crawling warns against continuing for more than one to two days. If the work will take longer, consider keeping a static, read-only copy of the site online instead.

Do not serve the maintenance page with a 200, and do not 302 everything to a single maintenance URL that returns 200. In both cases Google may index the maintenance text in place of your content. The reasoning for each status code is in the HTTP status codes guide.

Using errors on purpose to slow Googlebot

If Googlebot really is too much for the server, Google's documented emergency brake is to return 500, 503 or 429 to its requests for a short period. Apply it to the expensive URL pattern only if you can, and fix the cause, which is frequently an infinite URL space that belongs in robots.txt (see crawl budget explained). Check first that the traffic is real: much "Googlebot" load in logs comes from scrapers using its user agent.

After an outage: recovery

1. Confirm clean responses: spot check key URLs and watch the 5xx share in your logs fall to near zero.
2. Check Crawl stats host status returns to green over the following days.
3. Make sure your sitemap is reachable and accurate, so Google has a full list of URLs to revisit.
4. Use URL Inspection and Request indexing for a handful of top pages. The daily quota is small, so do not plan on using it for thousands of URLs.
5. Watch the Page indexing report. URLs that dropped out are usually reindexed as they are recrawled. If pages stay out for weeks after the server has been stable, look for other causes in why pages get deindexed.

Finally, set up monitoring that tells you before Google does: an uptime check on the homepage, a deep page and /robots.txt, plus an alert on the 5xx rate in your logs. TurboIndex tracks the index status of your URLs through the URL Inspection API, which shows you when pages start dropping out, but with server errors the log alert should always fire first.