Robots.txt Generator
Pick a preset, adjust the rules per crawler, add your sitemap and download a clean robots.txt. The preview updates while you type.
Start from a preset
Crawl-delay: Google ignores this directive. Bing and some other crawlers respect it. Leave it empty unless a specific bot overloads your server.
Live preview
robots.txt
User-agent: *
Disallow:
Upload the file to the root of the host, for example https://yoursite.com/robots.txt. Each subdomain needs its own file.
How robots.txt works
The file lives at the root of a host and is read by crawlers before they fetch anything else. It is made of groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules that apply to those crawlers.
A crawler uses only one group: the one that matches its name most specifically. Googlebot follows a User-agent: Googlebot group if it exists and ignores the * group in that case. That is the most common surprise: rules in * are not inherited by named groups.
Path matching
- Rules match from the start of the path.
Disallow: /cartalso blocks/cart/itemsand/cartoon. *matches any sequence of characters and$anchors the end:Disallow: /*.pdf$.- Paths are case sensitive.
- An empty
Disallow:blocks nothing.
What to block, and what to leave alone
Block URL spaces that waste crawling: internal search, cart and checkout, endless filter combinations, admin areas. Do not block CSS and JavaScript files, because Google renders pages and needs them. Do not use robots.txt to hide private content. The file is public and lists exactly the paths you would like people to ignore.
A single wrong line can take a site out of search, and Disallow: / left over from staging is the classic case. After you upload the file, test real URLs against it with the robots.txt tester, and check important pages with the indexability checker.
The sitemap line
The Sitemap: directive is independent of user agent groups and must be an absolute URL. You can list several. If you do not have a sitemap yet, create one with the sitemap generator. Once it is live, TurboIndex can read it daily and submit new pages to Google.
Questions and answers
Does Disallow remove a page from Google?
No. Disallow stops crawling, not indexing. A blocked URL can still appear in results without a description if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag or X-Robots-Tag header instead.
Does Google respect crawl-delay?
No. Googlebot ignores the crawl-delay directive and adjusts its crawl rate on its own, based on how your server responds. Bing and several other crawlers do respect it.
How do I block AI crawlers?
Add a group that names each bot and disallows everything. The preset covers GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended (Gemini training, separate from Google Search), PerplexityBot and Bytespider. Robots.txt is voluntary, so bots that ignore the standard need to be blocked at the server or CDN.
Does blocking Google-Extended affect my rankings?
Google states that Google-Extended does not affect inclusion or ranking in Google Search. It controls whether your content is used for Gemini models. Googlebot is a separate user agent.
Which rule wins when Allow and Disallow both match?
For Google, the most specific rule wins, which means the rule with the longest matching path. If both have the same length, the less restrictive Allow rule is used.