# Should you block GPTBot? AI training vs search crawlers explained

> What blocking GPTBot, ClaudeBot, Google-Extended and CCBot actually does, what it doesn't do, and how to decide. With a robots.txt for each choice.

Canonical page: https://getrankroot.com/guides/should-you-block-gptbot · Updated 2026-10-18

Short answer: blocking GPTBot tells OpenAI not to use your pages for training its models. It does not remove you from ChatGPT search, which uses a different crawler, OAI-SearchBot. So the real question isn't "AI or no AI", it's whether you want your content in future training data. For most product and marketing sites the answer is yes; for sites whose content is the product, it often isn't.

## Two jobs, different bots

AI companies separate crawlers by purpose, and your robots.txt can treat each one differently. The full list with user agents is on our [AI crawler list](https://getrankroot.com/ai-bots).

- Training: [GPTBot](https://getrankroot.com/ai-bots/gptbot) (OpenAI), [ClaudeBot](https://getrankroot.com/ai-bots/claudebot) (Anthropic), [Google-Extended](https://getrankroot.com/ai-bots/google-extended) (Google), [Applebot-Extended](https://getrankroot.com/ai-bots/applebot-extended) (Apple), [CCBot](https://getrankroot.com/ai-bots/ccbot) (Common Crawl) and [meta-externalagent](https://getrankroot.com/ai-bots/meta-externalagent) (Meta).
- Search and live answers: [OAI-SearchBot](https://getrankroot.com/ai-bots/oai-searchbot) and [ChatGPT-User](https://getrankroot.com/ai-bots/chatgpt-user) (ChatGPT), [Claude-SearchBot](https://getrankroot.com/ai-bots/claude-searchbot) and [Claude-User](https://getrankroot.com/ai-bots/claude-user) (Claude), [PerplexityBot](https://getrankroot.com/ai-bots/perplexitybot) and [Perplexity-User](https://getrankroot.com/ai-bots/perplexity-user) (Perplexity).

## What blocking each training crawler does

- GPTBot: OpenAI says disallowing GPTBot indicates your content should not be used to train its generative AI foundation models. ChatGPT search inclusion is controlled separately by OAI-SearchBot.
- ClaudeBot: Anthropic says disallowing it signals that your site's future materials should be excluded from its training datasets. Claude-SearchBot and Claude-User are separate.
- Google-Extended: not a separate crawler but a robots.txt token. Google says it controls whether content it crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI, and that it doesn't affect inclusion or ranking in Google Search.
- Applebot-Extended: Apple says it doesn't crawl; it only decides whether data Applebot collected may train Apple's foundation models. Pages that disallow it stay in Apple's search results, Spotlight, Siri and Safari.
- CCBot: keeps your future pages out of Common Crawl's open web archive, which many AI labs and researchers use as training data.

## What blocking does not do

- It doesn't remove anything a model has already learned. robots.txt applies to future crawling.
- It doesn't take you out of ChatGPT, Claude or Perplexity search, as long as their search and user-triggered bots are still allowed.
- It doesn't stop crawlers that ignore robots.txt. The major vendors document that their named crawlers follow it; for anything else you need firewall or CDN rules.
- Blocking Google-Extended doesn't remove you from AI Overviews. Google treats those as part of Search, controlled by Search's own snippet and indexing rules.

## Reasons to allow training

- When an assistant answers without searching, it relies on what the model learned. If your product isn't in that knowledge, it can't be recommended from memory.
- Your marketing pages, docs and pricing exist to be read. Having models describe them accurately is usually in your interest.
- It costs nothing extra. The same pages are already public and indexed by search engines.

## Reasons to block training

- Your content is what you sell: journalism, courses, original research, premium data.
- You're negotiating or planning to license your content.
- Legal, policy or client agreements require it.
- Crawl load is a problem. Anthropic supports the non-standard Crawl-delay directive if you'd rather slow its bots down than block them.

## What other sites do

In our [study of 538 SaaS and AI homepages](https://getrankroot.com/research/state-of-ai-visibility-2026), 4.1% blocked at least one AI training crawler and 0.6% blocked an AI search bot. Most software companies let both in, because being known to AI models is part of how they get found.

## robots.txt for each choice

Allow everything (most SaaS and marketing sites):

```
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml
```

Allow AI search, block training:

```
# AI search and user-triggered fetches: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Model training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml
```

Build a custom version with the [AI robots.txt generator](https://getrankroot.com/tools/robots-txt-generator), then confirm the result, including CDN and firewall blocks, with the [AI crawler checker](https://getrankroot.com/tools/ai-crawler-checker). Cloudflare and some hosts have "block AI bots" switches that can override robots.txt and block search bots too. For more on the file itself, see [robots.txt for AI crawlers](https://getrankroot.com/guides/robots-txt-ai-crawlers).

## Sources

- [OpenAI: Publishers and developers FAQ](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq)
- [OpenAI: Overview of OpenAI crawlers (read 2026-10-18)](https://developers.openai.com/api/docs/bots)
- [Anthropic: Does Anthropic crawl data from the web? (read 2026-10-18)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Google: List of Google's common crawlers (Google-Extended) (read 2026-10-18)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [Apple: About Applebot (read 2026-10-18)](https://support.apple.com/en-us/119829)
- [Perplexity: Perplexity crawlers (read 2026-10-18)](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- [Common Crawl: CCBot (read 2026-10-18)](https://commoncrawl.org/ccbot)

## FAQ

### Will blocking GPTBot remove my site from ChatGPT?

No. ChatGPT search uses OAI-SearchBot. OpenAI says the two settings are independent, and that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, so keep OAI-SearchBot allowed.

### Does blocking Google-Extended hurt my Google rankings?

No. Google says Google-Extended doesn't affect a site's inclusion in Google Search and isn't used as a ranking signal.

### Can I get my content removed from models that were already trained?

Not through robots.txt. It only affects future crawling. Check each vendor's documentation for any other opt-out or removal process.

### Do I need a separate rule for every subdomain?

Yes. robots.txt applies per host, so docs.yourdomain.com needs its own file. Anthropic specifically notes this for its crawlers.

## Check your site

Run the free audit at https://getrankroot.com/ or call `GET https://getrankroot.com/api/audit?url=example.com`.

## More guides

- [How to get your product recommended by ChatGPT, Perplexity and Claude](https://getrankroot.com/guides/how-to-get-recommended-by-chatgpt.md)
- [What is llms.txt? Format, examples and a free generator](https://getrankroot.com/guides/what-is-llms-txt.md)
- [robots.txt for AI crawlers: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot](https://getrankroot.com/guides/robots-txt-ai-crawlers.md)
- [Schema markup for AI search: the JSON-LD that helps AI describe you](https://getrankroot.com/guides/schema-markup-for-ai-search.md)
- [How to make your SaaS agent-ready (MCP, OpenAPI, llms.txt)](https://getrankroot.com/guides/make-your-saas-agent-ready.md)
- [How to track traffic from ChatGPT, Perplexity and Claude in GA4](https://getrankroot.com/guides/track-ai-traffic-ga4.md)
- [GEO vs SEO: what generative engine optimization actually changes](https://getrankroot.com/guides/geo-vs-seo.md)
- [How to check if ChatGPT mentions your brand (manual method + tools)](https://getrankroot.com/guides/check-if-chatgpt-mentions-your-brand.md)
- [llms.txt for WordPress: how to add it (manual, Yoast, plugins)](https://getrankroot.com/guides/llms-txt-wordpress.md)
- [Schema markup for SaaS: the JSON-LD worth adding for AI search](https://getrankroot.com/guides/schema-markup-for-saas.md)
- [GPTBot vs OAI-SearchBot vs ChatGPT-User: what to block](https://getrankroot.com/guides/gptbot-vs-oai-searchbot-vs-chatgpt-user.md)
- [ClaudeBot vs Claude-SearchBot vs Claude-User explained](https://getrankroot.com/guides/claudebot-vs-claude-searchbot-vs-claude-user.md)
- [What is an AI visibility score? (and how to check yours)](https://getrankroot.com/guides/what-is-an-ai-visibility-score.md)