Should you block GPTBot? AI training vs search crawlers explained

Short answer: blocking GPTBot tells OpenAI not to use your pages for training its models. It does not remove you from ChatGPT search, which uses a different crawler, OAI-SearchBot. So the real question isn't "AI or no AI", it's whether you want your content in future training data. For most product and marketing sites the answer is yes; for sites whose content is the product, it often isn't.

Two jobs, different bots

AI companies separate crawlers by purpose, and your robots.txt can treat each one differently. The full list with user agents is on our AI crawler list.

• Training: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Applebot-Extended (Apple), CCBot (Common Crawl) and meta-externalagent (Meta).
• Search and live answers: OAI-SearchBot and ChatGPT-User (ChatGPT), Claude-SearchBot and Claude-User (Claude), PerplexityBot and Perplexity-User (Perplexity).

What blocking each training crawler does

• GPTBot: OpenAI says disallowing GPTBot indicates your content should not be used to train its generative AI foundation models. ChatGPT search inclusion is controlled separately by OAI-SearchBot.
• ClaudeBot: Anthropic says disallowing it signals that your site's future materials should be excluded from its training datasets. Claude-SearchBot and Claude-User are separate.
• Google-Extended: not a separate crawler but a robots.txt token. Google says it controls whether content it crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI, and that it doesn't affect inclusion or ranking in Google Search.
• Applebot-Extended: Apple says it doesn't crawl; it only decides whether data Applebot collected may train Apple's foundation models. Pages that disallow it stay in Apple's search results, Spotlight, Siri and Safari.
• CCBot: keeps your future pages out of Common Crawl's open web archive, which many AI labs and researchers use as training data.

What blocking does not do

• It doesn't remove anything a model has already learned. robots.txt applies to future crawling.
• It doesn't take you out of ChatGPT, Claude or Perplexity search, as long as their search and user-triggered bots are still allowed.
• It doesn't stop crawlers that ignore robots.txt. The major vendors document that their named crawlers follow it; for anything else you need firewall or CDN rules.
• Blocking Google-Extended doesn't remove you from AI Overviews. Google treats those as part of Search, controlled by Search's own snippet and indexing rules.

Reasons to allow training

• When an assistant answers without searching, it relies on what the model learned. If your product isn't in that knowledge, it can't be recommended from memory.
• Your marketing pages, docs and pricing exist to be read. Having models describe them accurately is usually in your interest.
• It costs nothing extra. The same pages are already public and indexed by search engines.

Reasons to block training

• Your content is what you sell: journalism, courses, original research, premium data.
• You're negotiating or planning to license your content.
• Legal, policy or client agreements require it.
• Crawl load is a problem. Anthropic supports the non-standard Crawl-delay directive if you'd rather slow its bots down than block them.

What other sites do

In our study of 538 SaaS and AI homepages, 4.1% blocked at least one AI training crawler and 0.6% blocked an AI search bot. Most software companies let both in, because being known to AI models is part of how they get found.

robots.txt for each choice

Allow everything (most SaaS and marketing sites):

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Allow AI search, block training:

# AI search and user-triggered fetches: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Model training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Build a custom version with the AI robots.txt generator, then confirm the result, including CDN and firewall blocks, with the AI crawler checker. Cloudflare and some hosts have "block AI bots" switches that can override robots.txt and block search bots too. For more on the file itself, see robots.txt for AI crawlers.