AI crawler list: every AI bot and what it does

26 AI crawlers and robots.txt tokens from OpenAI, Anthropic, Google, Perplexity, Apple, Meta, Amazon, Microsoft, Mistral, DuckDuckGo and Common Crawl, taken from each company's own documentation. The key distinction: search bots get you cited, training bots feed models. Blocking the wrong one costs you traffic.

Jump to: quick robots.txt recipes · FAQ

Which AI bots can reach your site?

Reads your robots.txt and requests your page as each AI bot. Free, no signup.

AI search crawlers

Build the indexes AI assistants search and cite. Block these and you disappear from those answers.

User agent	Used for	Obeys robots.txt
`OAI-SearchBot` OpenAI	ChatGPT search	Yes
`Claude-SearchBot` Anthropic	Claude search	Yes
`PerplexityBot` Perplexity	Perplexity search	Yes
`Googlebot` Google	Google Search (incl. AI Overviews)	Yes
`Bingbot` Microsoft	Bing search and Copilot	Yes
`Applebot` Apple	Siri, Spotlight and Safari search	Yes
`meta-webindexer` Meta	Meta AI search	Yes
`Amzn-SearchBot` Amazon	Alexa and Amazon search	Yes
`DuckAssistBot` DuckDuckGo	DuckDuckGo AI answers	Yes
`MistralAI-Index` Mistral AI	Mistral search	Yes

User-triggered fetchers

Fetch a page live because a user asked about it. Several of them ignore robots.txt by design.

User agent	Used for	Obeys robots.txt
`ChatGPT-User` OpenAI	ChatGPT (user actions and GPT Actions)	Not reliably
`OAI-AdsBot` OpenAI	ChatGPT ads	n/a
`Claude-User` Anthropic	Claude (user requests)	Yes
`Perplexity-User` Perplexity	Perplexity (user requests)	Not reliably
`Google-CloudVertexBot` Google	Vertex AI Agents	Yes
`meta-externalfetcher` Meta	Meta AI (user requests)	Not reliably
`Amzn-User` Amazon	Alexa (user requests)	Not reliably
`MistralAI-User` Mistral AI	Mistral Vibe (user requests)	Yes

AI training crawlers

Collect content to train models. Blocking them is a policy choice and does not remove you from AI search.

User agent	Used for	Obeys robots.txt
`GPTBot` OpenAI	OpenAI model training	Yes
`ClaudeBot` Anthropic	Anthropic model training	Yes
`meta-externalagent` Meta	Meta AI training and indexing	Yes
`Amazonbot` Amazon	Amazon products and AI	Yes
`MistralAI-Training` Mistral AI	Mistral model training	Yes
`CCBot` Common Crawl	Common Crawl open dataset	Yes

Control tokens (do not crawl)

robots.txt tokens that never visit your site. They control how data another crawler already fetched may be used.

User agent	Used for	Obeys robots.txt
`Google-Extended` Google	Gemini training and grounding	n/a
`Applebot-Extended` Apple	Apple model training	n/a

Quick robots.txt recipes

Three common setups. Copy one into the robots.txt at your domain root, keep your Sitemap line, or build a custom file with the AI robots.txt generator.

1. Allow AI search, opt out of AI training (most sites)

# Allow AI search and user fetches
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Googlebot
User-agent: Google-CloudVertexBot
User-agent: Bingbot
User-agent: Applebot
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: MistralAI-Index
Allow: /

# Opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

2. Block all AI bots, stay in Google, Bing and Apple search

# Block AI crawlers, fetchers and training tokens
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: OAI-AdsBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: MistralAI-Training
User-agent: CCBot
Disallow: /

# Keep normal search engines
User-agent: *
Allow: /

Googlebot, Bingbot and Applebot stay allowed because blocking them removes you from those search engines. Google's AI Overviews and AI Mode use Google's Search index; limit them with nosnippet or max-snippet instead. User-triggered fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt at all.

3. Allow every AI crawler (maximum visibility)

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-AdsBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Google-CloudVertexBot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Applebot
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: meta-webindexer
Allow: /

User-agent: meta-externalfetcher
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: Amzn-SearchBot
Allow: /

User-agent: Amzn-User
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: MistralAI-User
Allow: /

User-agent: MistralAI-Index
Allow: /

User-agent: MistralAI-Training
Allow: /

User-agent: CCBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: *
Allow: /

Also check your CDN: Cloudflare and others have "block AI bots" settings that block requests before robots.txt matters, and Cloudflare changed its AI bot defaults on September 15, 2026. More in our robots.txt guide for AI crawlers.

FAQ

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot only collects training data. ChatGPT search uses OAI-SearchBot, and pages users ask about are fetched by ChatGPT-User. You can block GPTBot and still be cited in ChatGPT search.

Which AI bots ignore robots.txt?

User-triggered fetchers: OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores robots.txt, and Meta and Amazon say meta-externalfetcher and Amzn-User may not follow it. Anthropic says Claude-User does honor robots.txt.

How do I allow AI search but block AI training?

Allow the search and user bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot and others) and disallow the training bots and tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and others). Copy the ready-made robots.txt on this page.

Does Google-Extended affect Google Search or AI Overviews?

Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls Gemini training and grounding. AI Overviews and AI Mode are part of Google Search: Google says Googlebot's robots.txt rules are the crawl control, and nosnippet, max-snippet or noindex limit what is shown.

How do I block all AI bots but stay in Google and Bing?

Disallow the AI crawlers, user fetchers and training tokens by name and keep Googlebot, Bingbot and Applebot allowed, as in the block-all recipe on this page. Google's AI Overviews and AI Mode use Google's Search index, so to limit them use nosnippet, max-snippet or noindex rather than blocking Googlebot.

Sources: each bot page links to the operator's documentation. Last checked Oct 16, 2026. Spot an error? rankroot.support@proton.me