Skip to content
TurboIndex
Sign in

TurboIndexBot

You found this page because a request in your server logs pointed here. This is what made the request, why, and how to stop it.

Last updated: 11 October 2026

The user agent

Every request TurboIndex makes to a website identifies itself with this user agent string:

TurboIndexBot/1.0 (+https://turboindex.io/bot)

TurboIndexBot is not a search engine crawler. It does not follow links, it does not build an index of the web, and it does not come back on its own to sites nobody asked about. Every request has a person behind it: the owner of a connected site, or a visitor using a free tool.

Why it visited your site

There are two possible reasons.

1. Someone connected the site to TurboIndex

A person with access to the site's Google Search Console property added it to TurboIndex, a service that submits pages to Google and tracks whether they are indexed. For connected sites the bot fetches:

  • /robots.txt, to find Sitemap: lines and to read the rules for Googlebot,
  • the XML sitemaps and sitemap index files of the site, a few times per day, to notice new and changed URLs,
  • every page listed in those sitemaps, once before it is submitted to Google and again every 7 to 14 days (every 2 hours while a page does not answer), to check that it answers 200, does not redirect, carries no noindex, is not blocked by robots.txt and names no other URL as its canonical. Pages that fail this check are not submitted, so the owner's Google quota is not spent on them,
  • the IndexNow key file (/<key>.txt), if the owner turned on IndexNow, to confirm the key is in place.

The page checks are spread out: at most two requests at a time to one host, a few dozen pages of a site per minute at most, and never more than a set number of pages per site per day. A check is one plain GET with a 12 second limit; nothing on the page is executed. Index status itself still comes from Google's API, not from these requests.

2. Someone ran one of our free tools on a URL of your site

TurboIndex offers free SEO tools, such as an indexability checker, a redirect checker and a sitemap validator. When a visitor enters a URL, the bot fetches that URL once, and sometimes /robots.txt or a sitemap next to it, to produce the report. Anybody can enter any public URL, so the visitor may not be you.

How it behaves

  • Plain HTTP GET or HEAD requests. No JavaScript execution, no form submissions, no logins.
  • Response size and time are capped. A slow or very large response is abandoned.
  • A limited number of redirects is followed.
  • Requests to private and internal network addresses are refused.
  • The free tools are rate limited per visitor, so one person cannot use them to flood a site.

How to block it

Add a rule for the user agent token TurboIndexBot to your robots.txt. To block everything:

User-agent: TurboIndexBot
Disallow: /

To keep the bot away from one directory only:

User-agent: TurboIndexBot
Disallow: /private/

Keep in mind what a full block means if the site is yours and connected to TurboIndex: we can no longer read your sitemaps, so new pages are not picked up, and the page checks show every page as unreachable. In that case it is simpler to remove the site in the dashboard.

You can also block the user agent in your firewall or web server. That works for any bot, including this one. For nginx:

if ($http_user_agent ~* "TurboIndexBot") { return 403; }

Rules addressed to TurboIndexBot are always obeyed and are re-read every ten minutes. Generic User-agent: * groups are not applied, because each request is a single check that a person asked for, not a crawl. The indexability checker sends Googlebot's user agent string to see what Google would get, and it still obeys a TurboIndexBot block.

You can test your robots.txt rules with our robots.txt tester.

How to allow it

If your site is connected to TurboIndex and its pages show up as "Access denied" or "Unreachable" in the dashboard, a firewall, a CDN or a bot protection rule is most likely answering the bot with 403 or dropping the connection. Let requests through whose User-Agent header starts with TurboIndexBot/, the full string being:

TurboIndexBot/1.0 (+https://turboindex.io/bot)

In Cloudflare that is a WAF custom rule or a bot skip rule on the user agent (expression http.user_agent contains "TurboIndexBot", action Skip). In a web server, make sure no rule returns 403 for that user agent. The bot sends ordinary GET requests from TurboIndex's servers, needs no cookies, and asks for at most two pages at a time. A check that still fails three times in a row is shown as unreachable and the page is submitted to Google anyway, so a block costs you the check, not the submission.

Requests that only claim to be TurboIndexBot

A user agent string is easy to copy. If you see aggressive crawling, login attempts or form posts under this name, it is not us. Send a few log lines with timestamps and IP addresses to [email protected] and we will tell you whether the requests came from our servers.

Contact

Questions or a complaint about the bot: [email protected]. Tell us your domain and we will look at what was requested and why.