Index Monitoring: How to Catch Deindexed Pages Early

Index monitoring means keeping a list of the pages Google has indexed, rechecking each of them on a schedule, and raising an alert when one drops out, with the reason Google gives. You need it because deindexing is silent: Search Console does not email you when a page moves from "URL is on Google" to "URL is not on Google", the Page indexing report lags, and the first symptom most people notice is a traffic drop weeks later. A monitor built on the URL Inspection API catches the drop within a day or two and tells you whether it was a 404, a noindex, a redirect or Google's own decision.

Why indexed pages drop out

Being indexed is not a permanent state. Google recrawls indexed pages and decides again every time. The usual reasons a page goes from indexed to not indexed, roughly in the order I see them:

• A deploy pushed noindex. A staging setting, a plugin update or a template change adds <meta name="robots" content="noindex"> or an X-Robots-Tag header. Googlebot reads it on the next crawl and the page is out. The coverage state is "Excluded by 'noindex' tag".
• The page started redirecting. A URL change, a trailing slash rule or a language redirect sends Googlebot elsewhere. State: "Page with redirect". If the redirect target is wrong, you lose the page without a replacement.
• The page returns an error. A 404 after a content move, a 410, a 5xx that persists. Google retries 5xx for a while, then drops the page.
• Soft 404. The page returns 200 but the content now says "no results", "out of stock" or "nothing here". Google classifies it as a soft 404 and removes it.
• The canonical changed. A new canonical tag, a parameter, or a near duplicate elsewhere, and Google picks the other URL. States: "Alternate page with proper canonical tag" or "Duplicate, Google chose different canonical than user".
• Quality reassessment. After a core update, or after a site adds thousands of thin pages, Google becomes stricter about the whole host. Previously indexed pages land in "Crawled - currently not indexed".
• Hacked pages. Injected content or redirects served to Googlebot only. The page may stay indexed with the wrong content, or be removed, and a manual action can follow.
• Removal requests. Someone with Search Console access filed a temporary removal, or the page was blocked in robots.txt by mistake and later also lost its content.

The common thread: most of these are mistakes on your side that Google only reports after the fact, and all of them show up as a coverage state you can read per URL. The full list of causes is in why pages get deindexed.

Why you find out late

The Page indexing report lags. The chart updates every few days, and the "Indexed" count moves slowly on big sites. A page can be gone for a week before the report shows it, and you still have to open the report to see it. How the report is built is in the Page indexing report explained.

Search Console emails are about issues, not pages. You get a message when a new issue type appears or the count of an existing one jumps, and about sitemap errors and manual actions. You do not get one when page number 847 in your catalogue drops out.

The site: operator is useless for this. It returns an estimate, it misses indexed pages, and it keeps showing pages Google has already dropped. Do not build alerts on it.

Traffic lags more. A deindexed page stops getting impressions, but you only notice in the Performance report when the drop is big enough to stand out from normal variation, usually weeks later.

The only exact signal is URL Inspection, per URL. The tool in Search Console is manual; the API behind it is what a monitor uses. It returns the verdict, the coverage state, the last crawl time, the canonical Google chose and the robots.txt state, and allows 2,000 inspections per day and 600 per minute per property. How to read those fields is in the URL Inspection tool guide.

What a monitor has to do

A useful index monitor does five things.

1. Keep the inventory. The list of URLs you want indexed, usually your sitemaps. Pages that are not in the inventory are not monitored, which is a feature: feeds, filters and old junk should not be in it.
2. Record the state per URL. Indexed or not, the coverage state, the crawl date, the canonical. One row per URL, with history, so you can tell "dropped yesterday" from "never indexed".
3. Recheck on a schedule. Every indexed page gets inspected again, within the 2,000 per day budget. A site with 8,000 indexed pages can be covered every four days; a site with 500 can be checked daily and still have budget left for new pages.
4. Confirm before alerting. Search Console has glitches: the API sometimes returns a stale or odd verdict that corrects itself on the next call. A monitor should recheck a changed URL before it emails you, so a reporting hiccup does not become a Sunday evening panic.
5. Send the reason, not just the fact. "12 pages dropped out of the index" is a worry. "12 pages dropped: 9 now return 404, 3 are Excluded by noindex tag" is a to-do list.

Two things a monitor should not do: alert on every page that was never indexed (that is a different report), and resubmit a dropped page automatically without you looking at why. A page that dropped because it now says noindex should not be pushed back to Google.

Setting up free alerts

TurboIndex does the loop above at no cost: it reads the sitemaps submitted in Search Console, inspects each URL through the URL Inspection API, keeps the state and history per page, rechecks indexed pages on a schedule, and emails you when Google drops one, with the coverage state Google reported. Sign in with Google (the read-only Search Console scope), pick the property, and the first inspections start within the hour. New and changed pages found in the sitemap are also fetched by TurboIndexBot and submitted (see the crawler page for what it checks), but the monitoring works on its own if that is all you want.

Manual alternatives

If you would rather own the whole thing, three approaches work.

A weekly export diff

Open Indexing > Pages, click Export at the top right, and save the report, or open the "Indexed" table and export its rows (up to 1,000). Do it every week and compare with the previous file:

sort last-week.csv > a.txt
sort this-week.csv > b.txt
comm -23 a.txt b.txt   # URLs indexed last week and missing now

It is free and takes five minutes. The limits: 1,000 rows per table, the report lags, and you have to remember to do it.

A URL Inspection API script

Enable the Search Console API in a Google Cloud project, create a service account, add its email address as a user of the property under Settings > Users and permissions, and loop over your URLs:

from google.oauth2 import service_account
from googleapiclient.discovery import build

creds = service_account.Credentials.from_service_account_file(
    "sa.json", scopes=["https://www.googleapis.com/auth/webmasters.readonly"])
svc = build("searchconsole", "v1", credentials=creds)

def inspect(url, site="https://example.com/"):
    body = {"inspectionUrl": url, "siteUrl": site}
    r = svc.urlInspection().index().inspect(body=body).execute()
    s = r["inspectionResult"]["indexStatusResult"]
    return s["verdict"], s.get("coverageState"), s.get("lastCrawlTime")

Store the three fields in a database with a timestamp, run it daily within 2,000 calls, and email yourself when verdict changes from PASS to anything else for a URL that was PASS before. The confirmation step is one more call the next day.

Looker Studio

Google's Search Console connector for Looker Studio gives you Performance data (clicks and impressions per page), not the index state. It is good for spotting pages whose impressions fell to zero, which is the symptom, and for sharing a dashboard with a client. Pair it with one of the two methods above for the cause.

What to do when the alert arrives

Coverage state in the alert	Likely cause	First action
Not found (404)	Page deleted or URL changed	Restore the page or redirect to the replacement; remove it from the sitemap if it is gone for good
Excluded by 'noindex' tag	Deploy, plugin or template change	Remove the tag or header, confirm with the live test, request indexing once
Page with redirect	Redirect rule changed	Check the target with the redirect checker; fix the rule or update internal links and the sitemap
Soft 404	Content gone or an empty state served	Restore the content, or return a real 404 or 410
Server error (5xx)	Hosting or application fault	Fix the error, then wait; Google retries on its own
Alternate page with proper canonical tag	Canonical now points elsewhere	Decide which URL should win and make the tags, sitemap and links agree
Crawled - currently not indexed	Quality reassessment	Improve or consolidate the page; do not resubmit the same content
Blocked by robots.txt	robots.txt changed	Fix the rule and test it with the robots.txt tester

For a one-off check of a handful of URLs, the free Google index checker tells you which are on Google, with Google's own answer, after a Google sign-in. For a site you care about, put the recheck on a schedule, because the pages that drop out are rarely the ones you were looking at.