Bulk Google Index Checker: Check Many URLs Accurately

A bulk Google index checker only gives exact answers when it uses Google's URL Inspection API, which reports the index state of one URL per call at 2,000 calls per day per Search Console property. Everything else is an approximation: site: searches are a sample, the Page indexing report lags and caps its exports at 1,000 rows per table, and tools that check any domain without access are running site: searches for you. Below is what each method can and cannot tell you, a Python script for the API, and when a tool is the better choice.

Why site: is not a checker

site:yoursite.com/some-page looks like a yes or no question, and people build spreadsheets on it. Three problems:

1. It is a sample. The site: operator returns results Google chooses to show for that query, not a dump of the index. A page can be indexed and absent from the list, especially on large sites.
2. Canonicals hide pages. If Google indexed your URL under a different canonical (a parameter-free version, the www version, a duplicate elsewhere), a search for the exact URL shows nothing, yet the content is in the index.
3. Doing it at scale yourself does not work. Google answers bursts of automated site: queries with captchas, so scripts stall and spreadsheets fill with errors. Checkers that work for any domain run the searches through search data providers, which keeps them working, but their answers still carry the first two problems.

Use site: for a sanity check on a handful of URLs, with the understanding that "not shown" does not mean "not indexed".

Method 1: the Page indexing report and its export

Indexing > Pages in Search Console shows the total number of indexed pages and, below it, every reason why other URLs are not indexed, with a count per reason. Switch the top filter from "All known pages" to "All submitted pages", or pick one sitemap, and you get the status of exactly the URLs you care about.

Click a reason and you get a table of example URLs. Export gives you that table as CSV, Excel or Google Sheets, up to 1,000 rows. For a reason with 20,000 affected URLs you see 1,000 of them, and there is no page two.

What the report is good for: totals, trends, and spotting which reason is growing. What it is not: a per-URL list for a site with more than a thousand problem pages, or a current answer, because the data is refreshed in batches and runs days behind. Page indexing report explained goes through each reason.

One useful trick with no quota: Performance > Search results, Pages tab, exported. Every page that received an impression is indexed by definition. The export can run to tens of thousands of rows. Pages missing from it are "unknown", not "unindexed", since an indexed page with no impressions does not appear.

Method 2: the URL Inspection API

This is the only exact bulk source. It returns the same data as the URL Inspection tool, for one URL per request:

Field	What it tells you
verdict	PASS (indexed), NEUTRAL (excluded), FAIL (error)
coverageState	The exact words of the Page indexing report, such as "Submitted and indexed"
lastCrawlTime	When Googlebot last fetched the page; absent if never
robotsTxtState, indexingState, pageFetchState	Whether robots.txt, a noindex or a fetch error got in the way
googleCanonical, userCanonical	Which URL Google chose, and which you declared
referringUrls, sitemap	Where Google found the URL

Constraints: 2,000 inspections per day per property, 600 per minute, shared between everyone and every tool using the property. The siteUrl must be exactly as Search Console shows it (sc-domain:yoursite.com or https://www.yoursite.com/ with the slash). There is no live test through the API; you get the indexed state, which is what a checker needs.

This is also how every honest bulk checker works. The tool asks you to sign in with Google, lists your properties, and spends the property's quota on your behalf. If a tool checks a domain you have not verified, it is not using this API.

A Python script for the API

You need a Google Cloud project with the Search Console API enabled, a service account with a JSON key, and that service account's email added as a user on the Search Console property (Full permission works; the Owner role is only needed for the Indexing API). Put your URLs one per line in urls.txt.

# pip install google-auth requests
import csv
import time

from google.oauth2 import service_account
from google.auth.transport.requests import AuthorizedSession

SITE = "sc-domain:example.com"  # or "https://www.example.com/"
ENDPOINT = "https://searchconsole.googleapis.com/v1/urlInspection/index:inspect"
SCOPES = ["https://www.googleapis.com/auth/webmasters.readonly"]

creds = service_account.Credentials.from_service_account_file(
    "service-account.json", scopes=SCOPES
)
session = AuthorizedSession(creds)


def inspect(url):
    body = {"inspectionUrl": url, "siteUrl": SITE, "languageCode": "en-US"}
    r = session.post(ENDPOINT, json=body, timeout=60)
    if r.status_code == 429:
        return {"url": url, "verdict": "QUOTA"}
    r.raise_for_status()
    s = r.json()["inspectionResult"]["indexStatusResult"]
    return {
        "url": url,
        "verdict": s.get("verdict"),
        "coverageState": s.get("coverageState"),
        "lastCrawlTime": s.get("lastCrawlTime", ""),
        "googleCanonical": s.get("googleCanonical", ""),
    }


with open("urls.txt") as f:
    urls = [line.strip() for line in f if line.strip()]

fields = ["url", "verdict", "coverageState", "lastCrawlTime", "googleCanonical"]
with open("index-status.csv", "w", newline="") as out:
    writer = csv.DictWriter(out, fieldnames=fields)
    writer.writeheader()
    for url in urls[:2000]:
        row = inspect(url)
        writer.writerow(row)
        print(row["verdict"], url)
        if row["verdict"] == "QUOTA":
            break
        time.sleep(0.11)  # stays under 600 requests per minute

Notes on running it:

• The languageCode keeps coverageState in English so you can match on it.
• Stop on the first 429 and continue the next day. The daily counter is per property.
• verdict: NEUTRAL with "Crawled - currently not indexed" and NEUTRAL with "Discovered - currently not indexed" are different problems; keep the state text, not just the verdict.
• Keep yesterday's CSV and diff. A URL that went from PASS to NEUTRAL is the thing worth your attention.

Reading the output

coverageState	What it means	Usual next step
Submitted and indexed	Indexed	Nothing
Indexed, not submitted in sitemap	Indexed, but missing from your sitemap	Add it, or decide it should not be indexed
Discovered - currently not indexed	Known, never fetched	Internal links, request indexing for key pages
Crawled - currently not indexed	Fetched, declined	Improve or consolidate the content
Excluded by 'noindex' tag	You told Google not to	Remove the tag if unintended
Alternate page with proper canonical tag	Google kept the canonical instead	Nothing, unless the canonical is wrong
Not found (404), Soft 404, Page with redirect	The URL does not serve a page	Fix the URL or remove it from the sitemap
URL is unknown to Google	Never seen	Sitemap, links

The methods and their trade-offs for a single URL are compared in how to check if a page is indexed, and the tool's fields are explained in the URL Inspection tool guide.

Method 3: tools

For a handful of URLs of your own site, the free Google index checker takes a pasted list of up to 25 and runs URL Inspection for each one after a Google sign-in, so you get Google's own verdict and reason without a script. URLs of sites that are not in your Search Console get the site: search and URL Inspection links instead.

For whole sitemaps on a schedule, you want something that runs the API for you and keeps the history. TurboIndex does this after a Google sign-in with the read-only Search Console scope: it reads the sitemaps you submitted, inspects each URL within the property's daily quota, rechecks indexed pages later and emails you when Google drops one, with Google's stated reason next to each URL.

Be wary of any checker that sells certainty about a domain it has no access to. Without access, every checker is reading search results, and its "not indexed" can be wrong. Prefer one that tells you why a page may be missing and sends you to URL Inspection to confirm.

A routine for a large site

A 40,000-URL site cannot be inspected in one day on one property. Pattern that works:

1. Extract the URL list from your sitemaps with the sitemap URL extractor, and clean it with the URL list cleaner so that duplicates, fragments and non-canonical variants are gone before you spend quota.
2. Remove every URL that has an impression in the Performance export. Those are indexed and cost nothing to confirm.
3. Inspect the rest at 2,000 per day, newest pages first, since they are the ones most likely to be missing.
4. Once the backlog is done, inspect only new and changed URLs daily and resample the rest monthly.
5. If the site has several natural sections, verifying each as its own URL-prefix property gives each section its own 2,000 per day.

The next step after the checker says "not indexed" is the reason, and that comes from the same API response. Find all indexed pages covers the inverse question, building the complete list of what Google has.