Log File Analysis: See Exactly What Googlebot Crawls

Server logs record every request Googlebot makes to your site: which URL, when, which status code it got and how many bytes came back. Search Console shows you a sampled summary. Logs show you everything. With a verified set of Googlebot requests you can see where crawl activity is wasted, which sections are ignored, and which errors only crawlers run into.

What a log line contains

Most servers use the combined log format or something close to it:

66.249.66.1 - - [14/Sep/2026:06:25:13 +0000] "GET /products/boots?color=red HTTP/1.1" 200 18432 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

When you split a line on spaces, as awk does by default, the fields are:

awk field	Content
$1	Client IP
$4	Timestamp (with a leading [)
$6	Method (with a leading ")
$7	Requested path and query string
$9	Status code
$10	Response size in bytes
$12 onwards	User agent

Check your own format before trusting any command. Custom formats and extra fields shift the positions.

Default locations: /var/log/nginx/access.log and /var/log/apache2/access.log (or /var/log/httpd/access_log). Rotated files are usually gzipped. Use zcat or zgrep to read them without unpacking:

zcat -f /var/log/nginx/access.log* | grep "Googlebot" > googlebot.log

On managed hosting, look for a "raw access logs" option. On Cloudflare, Fastly, CloudFront and similar, log export is a feature of the CDN, sometimes on paid plans only.

Step 1: separate real Googlebot from impostors

The user agent string is trivial to fake, and plenty of scrapers use Googlebot's in the hope of being let through. If you analyse everything that claims to be Googlebot, your conclusions will be wrong. Google documents two ways to verify.

Reverse and forward DNS

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

1. Reverse lookup the IP from your logs.
2. Check that the hostname ends in googlebot.com, google.com or googleusercontent.com.
3. Forward lookup that hostname.
4. Check that it resolves back to the original IP.

The forward step matters. Anyone who controls the reverse DNS for their own IP range can make it say googlebot.com. Only Google can make crawl-66-249-66-1.googlebot.com resolve to a Google IP.

Do it in bulk for every IP that claims to be Googlebot:

grep "Googlebot" access.log | awk '{print $1}' | sort -u > ips.txt

while read ip; do
  name=$(host "$ip" | awk '/pointer/ {print $NF}' | sed 's/\.$//')
  case "$name" in
    *.googlebot.com|*.google.com|*.googleusercontent.com)
      if host "$name" | grep -q "$ip"; then
        echo "$ip" >> verified.txt
      else
        echo "$ip forward lookup mismatch ($name)" >> fake.txt
      fi ;;
    *) echo "$ip ${name:-no-ptr}" >> fake.txt ;;
  esac
done < ips.txt

grep -F -w -f verified.txt access.log | grep "Googlebot" > googlebot-verified.log

Published IP ranges

Google publishes JSON files with the IP ranges of its crawlers, in CIDR notation: one list for common crawlers such as Googlebot, one for special-case crawlers such as AdsBot, and lists for user-triggered fetchers. The file names and current locations are linked from Google's "Verifying Googlebot and other Google crawlers" documentation. Matching against the ranges is faster than DNS for large volumes, and it is what you should use for firewall allowlists. The lists change, so refresh them on a schedule.

Look at fake.txt before you move on. If a large share of "Googlebot" requests is fake, that traffic is a candidate for blocking or rate limiting, and it should never be counted as crawl activity.

All commands below assume you are working on googlebot-verified.log.

Step 2: the questions worth asking

How much does Googlebot crawl per day?

awk '{print substr($4,2,11)}' googlebot-verified.log | uniq -c

A stable line is normal. A sudden drop often lines up with server errors or a robots.txt change. A sudden spike often lines up with a newly exposed set of URLs, such as a filter that started generating crawlable links.

Which status codes does it get?

awk '{print $9}' googlebot-verified.log | sort | uniq -c | sort -rn

What you want to see is mostly 200 and 304. Things to chase:

• A high share of 301 or 302 means internal links or sitemaps point at redirecting URLs.
• A high share of 404 means something links to dead URLs, or an old URL pattern is still being retried. Some 404s are always present and harmless.
• Any notable share of 5xx or 429 needs attention right away, because it lowers crawl rate. See server errors and Googlebot.
• 403 for Googlebot on public pages points to a firewall or bot protection rule.

The meaning of each code for Google is in the HTTP status codes guide.

Which URLs return errors?

awk '$9 ~ /^(4|5)/ {print $9, $7}' googlebot-verified.log \
  | sort | uniq -c | sort -rn | head -50

Where does the crawl activity go?

By top level directory:

awk '{split($7, p, "/"); print "/" p[2]}' googlebot-verified.log \
  | sed 's/?.*//' | sort | uniq -c | sort -rn | head -20

Compare that with where your valuable pages are. If /products/ holds 80% of your indexable pages and receives 15% of the requests, while /search/ or /tag/ takes the rest, you have found your problem.

How much goes to parameter URLs?

# share of requests with a query string
awk '{ if ($7 ~ /\?/) q++; else c++ } END { printf "with query: %d\nclean: %d\n", q, c }' \
  googlebot-verified.log

# which parameters
grep -o '[?&][a-zA-Z0-9_]*=' googlebot-verified.log \
  | tr -d '?&=' | sort | uniq -c | sort -rn | head -20

Parameters such as sort, filter, sessionid or replytocom at the top of that list are crawl waste. Decide per parameter between a robots.txt disallow and a canonical tag. The trade-offs are in noindex vs robots.txt vs canonical.

Which pages are crawled most and least?

awk '{print $7}' googlebot-verified.log | sort | uniq -c | sort -rn | head -30

The top of this list is normally the homepage, robots.txt, sitemaps and hub pages. If it is a calendar widget or an API endpoint, investigate.

Which sitemap URLs were never crawled?

This is the most valuable check, and the one Search Console cannot do for you at URL level.

# paths Googlebot requested with a 200 response
awk '$9 == 200 {print $7}' googlebot-verified.log | sed 's/?.*//' | sort -u > crawled.txt

# paths from your sitemap (one URL per line in sitemap-urls.txt)
sed -E 's#https?://[^/]+##' sitemap-urls.txt | sort -u > wanted.txt

# in the sitemap, never crawled during the log period
comm -23 wanted.txt crawled.txt > never-crawled.txt
wc -l never-crawled.txt

You can produce sitemap-urls.txt with the Sitemap URL Extractor. Pages that are in the sitemap and have not been requested for a month or more are usually poorly linked internally, very deep in the site structure, or judged low priority by Google. They match what Search Console calls discovered, currently not indexed. Stronger internal linking is the usual first fix.

Run comm -13 for the opposite list: URLs that Googlebot crawls that are not in your sitemap. That is where you find old URL patterns, parameter variants and things you did not know existed.

Which Google crawler is it?

grep -o "Googlebot[-A-Za-z]*\|AdsBot-Google[-A-Za-z]*\|GoogleOther\|Storebot-Google" \
  googlebot-verified.log | sort | uniq -c | sort -rn

Most requests should come from Googlebot with a smartphone user agent (look for Android and Mobile in the string), since Google indexes the mobile version of pages. Requests from Googlebot-Image, AdsBot and others are separate crawlers with separate purposes.

Is the server slow for Googlebot?

Standard log formats do not include response time. Add it. In nginx:

log_format timed '$remote_addr - $remote_user [$time_local] "$request" '
                 '$status $body_bytes_sent "$http_referer" "$http_user_agent" '
                 '$request_time';
access_log /var/log/nginx/access.log timed;

In Apache, add %D (microseconds) to your LogFormat. Then average it per section. Slow responses lower Google's crawl capacity limit for your site, as described in crawl budget explained.

Step 3: turn findings into changes

Finding	Action
Many requests to parameter or filter URLs	Disallow the patterns in robots.txt, or canonicalise, and remove internal links to them
Many 301s	Update internal links and sitemaps to final URLs
Recurring 404s on one pattern	Find the linking source. Redirect if there is a real replacement, otherwise leave as 404
5xx in bursts at fixed times	Correlate with cron jobs, backups and deployments
403 or 429 for verified Googlebot	Fix firewall, WAF or rate limiter rules
Important pages never crawled	Add internal links from hub pages, confirm they are in the sitemap
Most "Googlebot" traffic is fake	Block or rate limit by verification, not by user agent
Requests to URLs you removed years ago	Normal. Google rechecks old URLs occasionally. Ignore if they return 404 or 410

After making changes, test that your new robots.txt rules match what you intended with the Robots.txt Tester, then compare the same log metrics two to four weeks later.

Practical notes

CDN in front of the origin. Cached responses never reach your server, so origin logs undercount. The origin may also log the CDN's IP. Configure your server to log the real client IP from the header your CDN sets (CF-Connecting-IP on Cloudflare, X-Forwarded-For elsewhere), and only trust that header when the connection actually comes from the CDN's address ranges.

Privacy. Access logs contain IP addresses of real visitors, which is personal data in many jurisdictions. Filter down to verified crawler requests early, and work with that extract instead of passing full logs around.

Size. For a few gigabytes, grep and awk are fine. Beyond that, load the verified extract into a database or a dedicated log analysis tool. The questions stay the same.

What logs cannot tell you. Logs show crawling. They do not show indexing. A URL that was crawled with a 200 may still not be indexed. Pair the log data with URL Inspection or the Page indexing report to see what Google did after the fetch.

Do it again. One analysis is a snapshot. The useful version is a small scheduled script that writes daily counts per status code and per section, so that a change in crawl behaviour is visible the week it happens.