How TurboIndex Checks Pages Before Submitting Them to Google

Before TurboIndex sends a URL to Google's Indexing API, it fetches that URL with its own user agent, TurboIndexBot, and reads the answer: the status code, any redirects, the robots meta tag, the X-Robots-Tag header, the canonical link, and signs of a soft 404. A page that answers 404 or 410, redirects, carries noindex or points its canonical elsewhere is never submitted, because Google could not index it anyway, and the page is shown in the dashboard as Blocked with the reason. A server error or a timeout is retried later, and a 403 or 429 is flagged but the page is sent, because bot protection usually blocks TurboIndexBot and not Googlebot. The point is simple: Google allows 200 submissions a day per project, and none of them should go to a page that cannot be indexed.

Why the check exists

The Indexing API quota is 200 publish requests per Google Cloud project per day, reset at midnight Pacific time. That is the whole budget for a site. Sitemaps, on the other hand, are full of URLs that should not be submitted: pages that moved and now redirect, pages deleted last month that still sit in a stale sitemap file, pages a plugin set to noindex, variants whose canonical points at the main version. Googlebot fetches such a page, finds the redirect or the noindex, and the request is gone.

On a site with a tidy sitemap the check changes little. On a site with 20,000 URLs, a few thousand of which are stale, it decides whether the day's 200 go to pages that can be indexed. It also surfaces problems early: a template that started emitting noindex after a theme update shows up as a block of Blocked pages the same day, not as a slow decline in the Page indexing report weeks later. How to spend the quota well in general is covered in Google Indexing API quota.

What TurboIndexBot fetches

The fetch is a plain HTTP GET with the user agent TurboIndexBot/1.0 (+https://turboindex.io/bot). No JavaScript is run, no forms are submitted, no cookies are kept. Redirects are followed up to a limit, each hop is recorded, and response size and time are capped, so a slow or enormous page is abandoned rather than waited for. Requests to private network addresses are refused. Rules addressed to TurboIndexBot in your robots.txt are obeyed on every fetch; generic User-agent: * rules are not applied, because each fetch is a single check for a page you asked about, not a crawl. Everything else about the bot, including how to block it, is on the /bot page and in the TurboIndexBot crawler guide.

What is checked

Check	Where it looks	Why it matters
Status code	The final HTTP response	Only a 200 can be indexed. 404 and 410 mean the page is gone
Redirect chain	Every 3xx hop and its target	Google indexes the destination, not the URL you submit
Robots meta tag	<meta name="robots"> in the HTML head	A noindex here tells Google to leave the page out
X-Robots-Tag	The HTTP response header	The same rule for PDFs, images and pages where the header is set by the server
Canonical	<link rel="canonical"> in the head	A canonical pointing at another URL asks Google to index that URL instead
Soft 404 signs	Title and body of a 200 response	A 200 page that says "not found" or has almost no content is treated by Google as an error page
Server errors and timeouts	5xx status, connection failures, the time cap	The page may be fine; the server is not, right now

What each result does

Result	Action	In the dashboard
200, no noindex, canonical to itself or none	Submitted	Queued, then Submitted
404 or 410	Never submitted	Blocked, with the reason
301, 302, 307, 308	Never submitted	Blocked, with the redirect target
noindex in the meta tag or the header	Never submitted	Blocked, with where the rule was found
Canonical pointing at a different URL	Never submitted	Blocked, with the canonical URL
Soft 404 signs	Flagged for you to look at	A warning next to the URL
500, 502, 503, 504, timeout, connection refused	Retried later	Stays in the queue
403 or 429	Submitted anyway, flagged	Submitted, with a flag that the fetch was refused

The reason stays visible, so you can go through the Blocked list once a week and fix the real problems: a sitemap still listing redirected URLs, a noindex nobody meant to set, a canonical copied from another template. Once the cause is fixed, select the URL and click Check status now so it is looked at again without waiting for the schedule.

The reasoning behind each decision

404 and 410. Google treats both as "gone". Submitting a URL that answers 404 asks Googlebot to come and confirm it is missing. If the page was deleted on purpose, remove it from the sitemap, which is where the URL came from. If it was not, you have found a broken page early.

Redirects. A submission for /old-page that redirects to /new-page is a request to crawl /old-page. Google follows the redirect and may index the target, but the URL you spent the request on will show as "Page with redirect". The sitemap should list /new-page, and then that URL is the one submitted. The redirect checker shows the whole chain for a URL, and the SEO side is in 301 vs 302 redirects.

noindex. A direct instruction to Google not to index the page, which a submission cannot override. The check reads both places the rule can live, the meta tag and the X-Robots-Tag header, because plugins set the first, servers and CDNs set the second, and people regularly forget one of them exists. The HTTP headers checker shows the header for any URL.

Canonical elsewhere. If /shoes?color=red declares /shoes as its canonical, Google will index /shoes. Submitting the variant spends a request on a page Google is asked to fold into another. If /shoes is in the sitemap, it is submitted on its own. When the canonical is wrong, which happens with copied templates and with sites that moved to https and kept the old canonical, fix the canonical rather than the submission; the canonical tags guide covers the usual mistakes.

Soft 404 signs. Google decides on its own whether a page is a soft 404, using signals nobody outside Google sees in full. So the check flags the clear cases, a 200 with "page not found" in the title or a nearly empty body, and leaves the decision to you. Common causes are a search page with no results, an out-of-stock product template, and a catch-all route that serves the app shell with a 200 for every path.

Server errors and timeouts. A 503 during a deploy says nothing about the page. The URL stays in the queue and is fetched again later. If the server keeps failing, the page stays out of submissions, which is the right outcome: Googlebot would get the same errors, and Google slows its crawling on hosts that answer with 5xx.

403 and 429. These two answers are usually about the fetcher, not the page. Cloudflare, Akamai, Sucuri and most hosting firewalls challenge or refuse unknown bots by default while allowing verified Googlebot through. Skipping refused pages would mean never submitting anything on a protected site, so the page is submitted and the result is flagged. The flag is your cue to allow the bot, so the check can do its job, or to confirm with the URL Inspection live test that Googlebot sees a normal page.

Allowing TurboIndexBot

If your pages come back as 403 or 429, allow the user agent where the block happens:

• Cloudflare. Security, WAF, Custom rules: add a rule with the expression (http.user_agent contains "TurboIndexBot") and the action Skip, and tick the features it should bypass. Check the Bots settings as well: the bot fight features block unverified bots on their own, and whether a skip rule can exempt them depends on your plan.
• nginx or Apache. Look for if ($http_user_agent ...) blocks, RewriteCond %{HTTP_USER_AGENT} lines or a bot blocklist include, and add an exception for TurboIndexBot.
• Security plugins on WordPress. Wordfence, Sucuri and similar have a user agent allowlist under their firewall settings.
• robots.txt. Only rules addressed to TurboIndexBot apply. If you have such a group with Disallow: /, the fetch is refused before it starts and the page cannot be checked. Remove the group or narrow it to the paths you really want kept away.

There is no published list of IP addresses at the moment, so the user agent is the thing to match.

How often pages are checked

The check runs right before each submission, so a page that is resubmitted after the cooldown (14 days on the free plan, at most three submissions in total) is fetched again at that point, and a redirect or noindex added in the meantime is caught then. A page that failed with a server error or a timeout is retried later without you doing anything. Robots.txt rules for the bot are re-read every ten minutes, so a change there takes effect quickly. Sitemaps are read every six hours, which is when new URLs enter the list and get their first check.

Index status is a separate matter and comes from Google, not from the fetch: the URL Inspection API is asked after 2, 3, 5, 7 and 14 days following a submission, and indexed pages are rechecked about every 30 days.

What the check cannot tell you

The fetch answers one question: is there anything on this page that makes a submission pointless? It does not predict what Google will do with a page that passes. "Crawled - currently not indexed" is Google's judgement about content and duplication, and a page can pass every check and still end there. The dashboard shows that verdict per URL after the inspection, with Google's own wording, and the fix is on the page, not in the submission.

It also does not render JavaScript. A page whose noindex is injected by a script, or whose canonical is rewritten client-side, passes the check and gets Google's rendered verdict later. For those cases the live test in URL Inspection is the tool. The indexability checker runs the same checks as the pre-check on any single URL, without signing in.

What to do next

1. Open the site's URL list, filter by Blocked, and read the reasons. Fix the sitemap for redirects and 404s, fix the page for noindex and canonical problems.
2. If you see fetches flagged as refused, allow TurboIndexBot in your firewall or CDN, and confirm with the URL Inspection live test that Googlebot gets a normal page in the meantime.
3. The check runs before every submission from now on; new problems show up as new Blocked rows.