# AI crawler list: every AI bot and what it does

> 26 AI crawlers and robots.txt tokens, taken from each company's own documentation. Search bots get you cited, training bots feed models.

Canonical page: https://getrankroot.com/ai-bots · Last checked Oct 16, 2026

## AI search crawlers

Build the indexes AI assistants search and cite. Block these and you disappear from those answers.

| User agent | Operator | Used for | Obeys robots.txt |
| --- | --- | --- | --- |
| [OAI-SearchBot](https://getrankroot.com/ai-bots/oai-searchbot.md) | OpenAI | ChatGPT search | Yes |
| [Claude-SearchBot](https://getrankroot.com/ai-bots/claude-searchbot.md) | Anthropic | Claude search | Yes |
| [PerplexityBot](https://getrankroot.com/ai-bots/perplexitybot.md) | Perplexity | Perplexity search | Yes |
| [Googlebot](https://getrankroot.com/ai-bots/googlebot.md) | Google | Google Search (incl. AI Overviews) | Yes |
| [Bingbot](https://getrankroot.com/ai-bots/bingbot.md) | Microsoft | Bing search and Copilot | Yes |
| [Applebot](https://getrankroot.com/ai-bots/applebot.md) | Apple | Siri, Spotlight and Safari search | Yes |
| [meta-webindexer](https://getrankroot.com/ai-bots/meta-webindexer.md) | Meta | Meta AI search | Yes |
| [Amzn-SearchBot](https://getrankroot.com/ai-bots/amzn-searchbot.md) | Amazon | Alexa and Amazon search | Yes |
| [DuckAssistBot](https://getrankroot.com/ai-bots/duckassistbot.md) | DuckDuckGo | DuckDuckGo AI answers | Yes |
| [MistralAI-Index](https://getrankroot.com/ai-bots/mistralai-index.md) | Mistral AI | Mistral search | Yes |

## User-triggered fetchers

Fetch a page live because a user asked about it. Several of them ignore robots.txt by design.

| User agent | Operator | Used for | Obeys robots.txt |
| --- | --- | --- | --- |
| [ChatGPT-User](https://getrankroot.com/ai-bots/chatgpt-user.md) | OpenAI | ChatGPT (user actions and GPT Actions) | Not reliably |
| [OAI-AdsBot](https://getrankroot.com/ai-bots/oai-adsbot.md) | OpenAI | ChatGPT ads | Not documented |
| [Claude-User](https://getrankroot.com/ai-bots/claude-user.md) | Anthropic | Claude (user requests) | Yes |
| [Perplexity-User](https://getrankroot.com/ai-bots/perplexity-user.md) | Perplexity | Perplexity (user requests) | Not reliably |
| [Google-CloudVertexBot](https://getrankroot.com/ai-bots/google-cloudvertexbot.md) | Google | Vertex AI Agents | Yes |
| [meta-externalfetcher](https://getrankroot.com/ai-bots/meta-externalfetcher.md) | Meta | Meta AI (user requests) | Not reliably |
| [Amzn-User](https://getrankroot.com/ai-bots/amzn-user.md) | Amazon | Alexa (user requests) | Not reliably |
| [MistralAI-User](https://getrankroot.com/ai-bots/mistralai-user.md) | Mistral AI | Mistral Vibe (user requests) | Yes |

## AI training crawlers

Collect content to train models. Blocking them is a policy choice and does not remove you from AI search.

| User agent | Operator | Used for | Obeys robots.txt |
| --- | --- | --- | --- |
| [GPTBot](https://getrankroot.com/ai-bots/gptbot.md) | OpenAI | OpenAI model training | Yes |
| [ClaudeBot](https://getrankroot.com/ai-bots/claudebot.md) | Anthropic | Anthropic model training | Yes |
| [meta-externalagent](https://getrankroot.com/ai-bots/meta-externalagent.md) | Meta | Meta AI training and indexing | Yes |
| [Amazonbot](https://getrankroot.com/ai-bots/amazonbot.md) | Amazon | Amazon products and AI | Yes |
| [MistralAI-Training](https://getrankroot.com/ai-bots/mistralai-training.md) | Mistral AI | Mistral model training | Yes |
| [CCBot](https://getrankroot.com/ai-bots/ccbot.md) | Common Crawl | Common Crawl open dataset | Yes |

## Control tokens (do not crawl)

robots.txt tokens that never visit your site. They control how data another crawler already fetched may be used.

| User agent | Operator | Used for | Obeys robots.txt |
| --- | --- | --- | --- |
| [Google-Extended](https://getrankroot.com/ai-bots/google-extended.md) | Google | Gemini training and grounding | Token only |
| [Applebot-Extended](https://getrankroot.com/ai-bots/applebot-extended.md) | Apple | Apple model training | Token only |

## Quick robots.txt recipes

### 1. Allow AI search, opt out of AI training (most sites)

```
# Allow AI search and user fetches
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Googlebot
User-agent: Google-CloudVertexBot
User-agent: Bingbot
User-agent: Applebot
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: MistralAI-Index
Allow: /

# Opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
```

### 2. Block all AI bots, stay in Google, Bing and Apple search

```
# Block AI crawlers, fetchers and training tokens
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: OAI-AdsBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: MistralAI-Training
User-agent: CCBot
Disallow: /

# Keep normal search engines
User-agent: *
Allow: /
```

Googlebot, Bingbot and Applebot stay allowed because blocking them removes you from those search engines. User-triggered fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt at all.

### 3. Allow every AI crawler (maximum visibility)

```
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-AdsBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Google-CloudVertexBot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Applebot
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: meta-webindexer
Allow: /

User-agent: meta-externalfetcher
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: Amzn-SearchBot
Allow: /

User-agent: Amzn-User
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: MistralAI-User
Allow: /

User-agent: MistralAI-Index
Allow: /

User-agent: MistralAI-Training
Allow: /

User-agent: CCBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: *
Allow: /
```

Also check your CDN: Cloudflare and others have "block AI bots" settings that block requests before robots.txt matters. Check a live site with https://getrankroot.com/tools/ai-crawler-checker or `GET https://getrankroot.com/api/bots?url=example.com`.

## FAQ

### Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot only collects training data. ChatGPT search uses OAI-SearchBot, and pages users ask about are fetched by ChatGPT-User. You can block GPTBot and still be cited in ChatGPT search.

### Which AI bots ignore robots.txt?

User-triggered fetchers: OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores robots.txt, and Meta and Amazon say meta-externalfetcher and Amzn-User may not follow it. Anthropic says Claude-User does honor robots.txt.

### How do I allow AI search but block AI training?

Allow the search and user bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot and others) and disallow the training bots and tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and others). Copy the ready-made robots.txt on this page.

### Does Google-Extended affect Google Search or AI Overviews?

Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls Gemini training and grounding. AI Overviews and AI Mode are part of Google Search: Google says Googlebot's robots.txt rules are the crawl control, and nosnippet, max-snippet or noindex limit what is shown.

### How do I block all AI bots but stay in Google and Bing?

Disallow the AI crawlers, user fetchers and training tokens by name and keep Googlebot, Bingbot and Applebot allowed, as in the block-all recipe on this page. Google's AI Overviews and AI Mode use Google's Search index, so to limit them use nosnippet, max-snippet or noindex rather than blocking Googlebot.