Your site is crawlable when Googlebot can request any page you want indexed and get the real content back with a 200 status, without being blocked by robots.txt, challenged by bot protection, sent through broken redirects, or shown a login or an empty shell. The fastest check is a live test in Search Console on a few representative URLs; the thorough check is the list below, which covers every layer between Googlebot and your HTML. Run it once when a site launches and again after every hosting, CDN or framework change.
Start with the live test
In Search Console, paste a URL into the inspection box and click Test live URL. Google fetches the page as the smartphone Googlebot, right now, and reports:
- Crawl allowed? Yes or no, from robots.txt.
- Page fetch: successful, failed (with the error), or blocked.
- Indexing allowed? Yes or no, from meta robots and X-Robots-Tag.
- Crawled as: which Googlebot. It is the smartphone agent for all sites now.
- View tested page: the rendered HTML, a screenshot, and under More info, the page resources that failed to load.
Test the home page, one page of each template (article, product, category) and one deep page. If all pass with the content visible in the screenshot, the site is crawlable; the checklist then only has to explain the ones that fail.
The checklist
1. robots.txt
Googlebot fetches /robots.txt on every host (protocol and subdomain each count as a host) before crawling it. A Disallow that matches a page stops the crawl of that page. Common accidents: Disallow: / left from staging, a rule blocking /wp-content/ or /assets/ that takes CSS and JavaScript with it, and a robots.txt that answers 5xx, which makes Google hold off crawling the whole site.
Test a URL against the live file with the robots.txt tester, and look at Search Console's robots.txt report under Settings for what Google last fetched. Syntax and examples are in the robots.txt guide.
2. Meta robots and X-Robots-Tag
A noindex does not stop crawling, but it stops indexing, and nofollow stops Google from following the page's links, which can cut off crawl paths. Check the <head> and the response headers of each template. Headers are easy to miss: a CDN or a staging rule can add X-Robots-Tag: noindex to every response without touching the HTML.
3. HTTP status codes
Googlebot wants a 200. What else it gets matters:
| Status | Effect on crawling |
|---|---|
| 301 or 302 | Followed, up to 10 hops |
| 401 or 403 | Page not crawled; Google treats the content as unavailable |
| 404 or 410 | Page dropped from the index once seen |
| 429 | Retried later, crawl rate reduced |
| 5xx | Retried later; persistent errors slow crawling of the whole host |
Run your important URLs through the HTTP status checker and compare with what the live test reports. A difference between the two usually means a rule that treats bots differently from browsers. The full table is in HTTP status codes for SEO.
4. Redirects
Chains (http to https to www to a trailing slash) cost time and, past 10 hops, stop Googlebot. Loops stop it immediately. Redirects that depend on the visitor's country or language send Googlebot, which mostly crawls from the United States, to one version only. Check that every URL in the sitemap answers 200 directly, with no hop at all.
5. JavaScript rendering
Googlebot renders JavaScript, but later, with a budget, and only what loads without interaction. If the main content arrives through an API call that fails for Google, sits behind a click, or depends on a script blocked by robots.txt, the rendered page is empty. The live test's screenshot and HTML tab show what Google got. The fixes are in JavaScript SEO: rendering and indexing.
6. Blocked resources
Open More info > Page resources in the live test. CSS and JavaScript files blocked by robots.txt or hosted on a domain that refuses Googlebot change how the page renders and can hide content and links. Allow them; there is no benefit in blocking assets.
7. Bot protection
Cloudflare, Akamai, AWS WAF, Imperva and the "under attack" modes of smaller hosts sit in front of your server and decide who gets through. Verified Googlebot is normally allowed, but a custom rule, a rate limit, a challenge applied site-wide, or a block on a hosting provider's IP range can serve Googlebot a challenge page or a 403. The symptom is a live test that fails to fetch, or coverage states such as "Blocked due to access forbidden (403)" and "Server error (5xx)" with no matching error in your own logs, because the request never reached your server.
Check the protection layer's own logs or analytics for requests from Googlebot's IP ranges that were challenged or blocked, and allow by verified bot, not by user agent string.
8. Slow responses and timeouts
Googlebot gives up on pages that take too long and reduces how fast it crawls a host that responds slowly or with errors. The Crawl stats report in Search Console (Settings > Crawl stats) shows average response time and the share of requests that ended in errors over the last 90 days. If response time climbs when traffic peaks, Googlebot is being served at the worst moment; fix caching or capacity before anything else.
9. Geo blocking
Country blocks that return 403, a challenge or a "not available in your region" page to visitors outside your market also hit Googlebot, which crawls from the United States for most sites. If your content is only for one country, serve it to everyone and let search engines decide where to rank it; blocking Google's crawl location means blocking Google.
10. Login walls, cookie walls and interstitials
Anything the visitor must click through before the content appears is a problem if the content is not in the HTML underneath. A login that redirects to a form means the page is not crawlable at all. A cookie consent screen that hides the content until accepted leaves Google looking at the consent screen; serve the content and overlay the notice. Age gates and newsletter interstitials follow the same rule.
11. Mobile parity
Google crawls with the smartphone Googlebot for all sites. Content and links that only exist in the desktop layout (a mega menu replaced by a short mobile menu, a sidebar removed on small screens) are invisible to it. Test with a phone-sized viewport and compare the links on the page with the desktop version.
Testing from the command line, and why it is not enough
A quick first check:
curl -sI -A "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/some-page
It shows the status code, the redirect target and the X-Robots-Tag header as served to a request that claims to be Googlebot. It cannot show what Google sees, because the serious protection layers verify the bot by IP address, not by user agent; your laptop's IP with Googlebot's user agent can be treated as a spoofing attempt and challenged, or allowed because the rule only looks at the string. Use curl to find obvious mistakes, then confirm with the live test.
Reading your logs
Server logs settle arguments. Filter for the Googlebot user agent, then verify a sample of IPs, because anyone can send that string. Google describes two ways:
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1
The reverse lookup must land on googlebot.com, google.com or googleusercontent.com, and the forward lookup must return the original IP. For automation, Google publishes its crawler IP ranges as JSON files (common-crawlers.json for Googlebot, plus files for special crawlers and user-triggered fetchers) that a firewall can load. With verified requests in hand, look at the status codes they got, which URLs they asked for and how often. A guide to that analysis is in log file analysis for Googlebot.
Crawlable is not indexable
Clearing every item above means Google can fetch your pages. Whether it indexes them is a separate decision based on content, duplication and the signals it collects, which the Page indexing report explains per URL. The order is fixed, though: nothing gets indexed that could not be crawled, so this checklist comes first.
If you want the fetch layer checked continuously rather than once, TurboIndexBot does it as part of TurboIndex: every URL in your sitemaps is fetched before anything is submitted to Google, and the result per URL (status, redirect target, robots directives, canonical, soft 404 signs) is shown in the dashboard, so a protection rule that starts blocking bots shows up the day it happens. The crawler is itself a bot, so the same protection that blocks Googlebot can block it; the user agent and how to allow it are on the crawler page.
Frequently asked questions
- What is the quickest way to check if Google can crawl my site?
- Inspect a URL in Search Console and click Test live URL. The result shows whether crawling is allowed by robots.txt, whether the page could be fetched, whether indexing is allowed, and which resources were blocked. It is a real fetch by Google, which is what matters.
- Is crawlable the same as indexable?
- No. Crawlable means Googlebot can fetch the page. Indexable means Google is allowed to put it in the index and chooses to. A page can be crawlable but noindexed, and a page can be blocked from crawling yet indexed from links without content. Check both.
- Can I just curl my site with a Googlebot user agent?
- It is a useful first test for robots rules and status codes, but it is not what Google sees. Bot protection services verify Googlebot by IP address, so a request with the user agent from your laptop may be challenged or allowed differently. The live test in Search Console and your server logs tell you what happened to real Googlebot requests.
- Does Cloudflare block Googlebot?
- Not by default: verified search crawlers are allowed through. Custom firewall rules, rate limits, a country block or a managed challenge applied to the whole site can still catch it. If the live test reports a fetch error or you see 403 or challenge pages for Googlebot in your logs, review those rules.