TurboIndexBot: What It Is and How to Block It

TurboIndexBot is the user agent TurboIndex uses whenever it has to fetch something from a website: the robots.txt and sitemaps of a site connected to the service, each page before it is submitted to Google, the IndexNow key file when that is turned on, and the single URL a visitor enters into one of the free SEO tools. It is not a search engine crawler. It follows no links, builds no index and never shows up on a site that nobody connected or asked about. You can block it with one group in robots.txt, and a block is obeyed on every request.

This page is the long version of /bot, for site owners who found the user agent in their logs and for TurboIndex users who want to know what the bot does on their behalf.

The user agent string

Every request identifies itself with:

TurboIndexBot/1.0 (+https://turboindex.io/bot)

The URL in the string is the standard way for a bot to say where its documentation is. If you see this string with any other form, or with a different URL, it is not from TurboIndex.

What it fetches, and why

There are two reasons the bot can appear in a log.

1. The site is connected to TurboIndex

Someone with access to the site's Search Console property added the site. For a connected site the bot fetches:

• /robots.txt, to find Sitemap: lines when no sitemap is submitted in Search Console, and to read any rules addressed to TurboIndexBot.
• The XML sitemaps and sitemap index files, to notice new and changed URLs. These are re-read every six hours.
• Each page, once, before it is submitted to Google. This is the pre-check: the fetch records the status code, the redirect chain, the robots meta tag, the X-Robots-Tag header, the canonical link and signs of a soft 404. A page that answers 404 or 410, redirects, carries noindex or has a canonical pointing elsewhere is never submitted; a server error or timeout is retried later; a 403 or 429 is flagged but submitted anyway, because bot protection often refuses this bot and not Googlebot. The full logic is in how TurboIndex checks pages before submitting.
• The IndexNow key file (/<key>.txt), once, when the owner turns on IndexNow, to confirm the file is in place.

Index status never comes from these fetches. It comes from Google's URL Inspection API, so the bot does not crawl a connected site to see what is indexed.

2. Someone ran a free tool on a URL of the site

TurboIndex offers free tools such as an indexability checker, a redirect checker, an HTTP headers checker and a sitemap validator. When a visitor enters a URL, the bot fetches that URL once, and sometimes /robots.txt or a sitemap next to it, to produce the report. Anybody can enter any public URL, so the visitor may not be you. The tools are rate limited per visitor, so one person cannot use them to send a stream of requests at a site.

How often it comes

What	How often
robots.txt	When a sitemap or page fetch needs it; rules for the bot are cached for ten minutes
Sitemaps of a connected site	Every six hours, or when the owner clicks Read sitemaps now
A page of a connected site	Once before each submission; a submission is repeated at most three times, after a cooldown of 14 days on the free plan
A page that answered with a server error or timed out	Retried later
IndexNow key file	Once at setup, and again when the owner clicks Check again
A URL entered in a free tool	Once per tool run, rate limited per visitor

Put together, a connected site with 1,000 pages sees a few sitemap fetches a day and up to about 200 page fetches a day, one per submission (no plan sends more than 200 Google submissions for one site a day, and the free plan sends 20), plus the retries of pages that answered with a server error. There is no crawl-delay support, because there is nothing to delay: the bot does not walk a site.

How it behaves

• Plain HTTP GET or HEAD requests. No JavaScript execution, no form submissions, no logins, no cookies carried between requests.
• Redirects are followed up to a limit, and every hop is checked before it is requested.
• Response size and time are capped. A slow or very large response is abandoned.
• Requests to private and internal network addresses are refused outright, so the bot cannot be pointed at something behind your firewall by way of a public hostname.
• It reads the page to produce the check result and keeps only that result.

Where requests come from, and how to verify one

TurboIndex does not publish a list of IP addresses at the moment, so the reliable way to verify a request is to ask. Send a few log lines with timestamps and IP addresses to [email protected] and you will get a yes or no.

What you can check yourself, from the behaviour:

• A real request is a single GET of a page, a sitemap, robots.txt or a key file. Dozens of requests per second, login attempts, POST requests or fetches of admin paths are not from TurboIndex.
• A real request obeys a TurboIndexBot group in robots.txt within ten minutes. If you add Disallow: / and the requests continue the next day, they are not from TurboIndex.
• A user agent string is trivial to copy. The same reasoning you apply to a fake Googlebot applies here; the reverse DNS and forward DNS technique in log file analysis for Googlebot does not have an equivalent for TurboIndexBot yet, which is why the email route exists.

How to block it, or limit it

In robots.txt

The bot obeys rules addressed to its own name. A generic User-agent: * group is not applied, because each request is a single check that a person asked for, not a crawl. To block everything:

User-agent: TurboIndexBot
Disallow: /

To keep it out of one directory only:

User-agent: TurboIndexBot
Disallow: /private/

Allow rules and the * and $ wildcards work the way they do for Googlebot; the longest matching rule wins. Test your file with the robots.txt tester, which evaluates a URL against a chosen user agent, and if you are writing the file from scratch the robots.txt generator produces a valid one. The general syntax is in the robots.txt guide.

In the web server or firewall

This works for any bot, including this one. nginx:

if ($http_user_agent ~* "TurboIndexBot") { return 403; }

Apache, in the virtual host or .htaccess:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} TurboIndexBot [NC]
RewriteRule .* - [F]

On Cloudflare, a WAF custom rule with the expression (http.user_agent contains "TurboIndexBot") and the action Block does the same. The patterns for blocking everything, or everything except a few crawlers, are in robots.txt: block all crawlers.

What a block means if the site is yours

If the site is connected to TurboIndex and you block the bot, three things stop working:

• Sitemaps can no longer be read, so new pages are not picked up.
• Pages can no longer be pre-checked before submission, so there is no way to keep a 404, a redirect or a noindex page out of the day's 200 submissions.
• The IndexNow key file cannot be verified.

If the aim is to stop TurboIndex working on the site, it is simpler to remove the site in the dashboard, and to remove the indexing account under Settings, Users and permissions in Search Console if you also want to revoke the Indexing API access. If the aim is to keep the bot out of one section of the site, a Disallow for that path does it without touching the rest.

If a firewall is returning 403 to the bot while you want the pre-check to work, allow the user agent there. The dashboard flags pages whose fetch was refused, so you can see when that is happening.

What it never does

• It never follows links from one page to another. The only URLs it requests are the ones in your sitemaps, the ones a TurboIndex user added by hand, and the ones a visitor typed into a tool.
• It never fetches the pages of a connected site in bulk to see what is indexed. That answer comes from Google's API.
• It never comes back to a site after the site is removed from TurboIndex, except when a visitor runs a free tool on one of its URLs.
• It never runs JavaScript, submits forms, signs in, or sends anything other than GET and HEAD.
• It never reaches private networks, even when a public hostname resolves to a private address.

Privacy

The bot keeps the result of each check, not the content: status code, redirect hops and their targets, the robots directives it found, the canonical URL, a flag for soft 404 signs, response time and the time of the check. That is what the dashboard shows next to each URL and what the free tools show in their reports. Page bodies are read to compute those fields and are not stored. The TurboIndex privacy policy lists what the service as a whole stores and for how long.

Contact

Questions, a complaint about the bot, or requests that only claim to be from it: [email protected]. Include your domain and, for suspected fake requests, a few log lines with timestamps and IP addresses, and you will be told what was requested and why, or that the requests did not come from TurboIndex.