Free tools · runs in your browser · no signup
AI robots.txt generator
Pick which of 26 AI crawlers may read your site. Start from a preset, fine-tune per bot, then copy your robots.txt.
Recommended for most sites: you stay citable in AI answers but opt out of model training.
AI search and answer crawlers
Build the indexes AI assistants search and cite. Blocking one removes you from that assistant's answers.
User-triggered fetchers
Fetch a page live because a user asked about it. Several say they generally ignore robots.txt, so a rule here has little effect for those.
AI training crawlers and tokens
Collect content to train models, or (control tokens) govern whether data another crawler fetched may be used for training. Blocking them does not remove you from AI search.
# robots.txt generated with https://getrankroot.com/tools/robots-txt-generator # Allowed AI crawlers User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: OAI-AdsBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Googlebot User-agent: Google-CloudVertexBot User-agent: Bingbot User-agent: Applebot User-agent: meta-webindexer User-agent: meta-externalfetcher User-agent: Amzn-SearchBot User-agent: Amzn-User User-agent: DuckAssistBot User-agent: MistralAI-User User-agent: MistralAI-Index Allow: / # Blocked AI crawlers User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot User-agent: MistralAI-Training User-agent: CCBot Disallow: / # Everyone else User-agent: * Allow: /
Running more than one page? The RankRoot Traffic Boost audits your full site and generates fix files for every page, for $49 once.
Training bots vs. search bots
Most AI companies run separate bots for separate jobs, and robots.txt lets you treat them differently. The distinction that matters for traffic:
- Search and answer crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, Amzn-SearchBot, MistralAI-Index, DuckAssistBot) build the indexes that AI assistants search and cite. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. Block these and you lose citations.
- User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User and others) visit a page because someone asked about it. Several say they generally ignore robots.txt for that reason.
- Training crawlers (GPTBot, ClaudeBot, meta-externalagent, Amazonbot, MistralAI-Training, CCBot) collect content to train models. Blocking them is a policy choice and, per OpenAI and Anthropic, does not remove you from ChatGPT or Claude search.
- Control tokens (Google-Extended, Applebot-Extended) never visit your site. They tell Google and Apple whether content their normal crawlers fetched may be used for AI training. Google and Apple say they do not affect regular search.
That is why "Allow AI search, block training" is the default here. Full details for each bot are in the AI crawler list. After you publish, confirm nothing else blocks them with the AI crawler checker: CDN and firewall rules often block bots robots.txt allows.
FAQ
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot collects content that may be used to train OpenAI's models. ChatGPT search uses OAI-SearchBot, and pages users ask about are fetched by ChatGPT-User. You can block GPTBot and still be cited in ChatGPT search.
Does blocking ClaudeBot remove me from Claude?
No. Anthropic says ClaudeBot collects training data, Claude-SearchBot indexes pages for Claude's search results and Claude-User fetches pages when a user asks. Blocking ClaudeBot does not affect the other two.
Does Google-Extended affect Google Search?
Google says Google-Extended does not affect inclusion or ranking in Google Search. It is a robots.txt token, not a crawler, and controls whether content Google already crawls may be used for Gemini training and grounding.
Will every AI bot obey my robots.txt?
No. OpenAI, Perplexity, Meta and Amazon say their user-triggered fetchers (ChatGPT-User, Perplexity-User, meta-externalfetcher, Amzn-User) may not follow robots.txt because a person asked for the page. Anthropic says Claude-User does honor it. Firewall and CDN rules can also block bots your robots.txt allows.
How long until changes take effect?
It depends on the operator. OpenAI says about 24 hours, DuckDuckGo says about 72 hours, and Amazon says its crawlers cache robots.txt for up to 30 days.
More free AI visibility tools
AI crawler checker
See which AI bots (OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot and more) can reach your site, including CDN and firewall blocks.
llms.txt generator
Paste your URL and get a ready-to-publish llms.txt built from your homepage, plus AI-friendly robots.txt and schema.
Schema markup generator (JSON-LD)
Build valid JSON-LD for Organization, SoftwareApplication, Product, FAQPage and WebSite. Fill in a form, copy the script tag. Free, runs in your browser.
FAQ schema generator
Paste questions and answers, get an HTML FAQ section plus FAQPage JSON-LD that AI answer engines and search can read. Free, no signup.
llms.txt validator
Check an llms.txt against the llmstxt.org format by URL or by pasting it: H1, summary, H2 link lists and the Optional section, with line-by-line fixes.
AI crawler log analyzer
Paste your server access logs and see which AI bots (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and more) crawled your site, which pages they hit and what status codes they got.
RankRoot Traffic Boost
Full-site audits and whole-site fix files: robots.txt, llms.txt and schema for every page. $49 once, no subscription.