Anthropic runs three bots with separate jobs. ClaudeBot collects public web content that may be used to train Anthropic's models. Claude-SearchBot indexes content for Claude's search responses. Claude-User fetches a page when a Claude user asks a question that needs it. Anthropic says its bots honor robots.txt, so you can opt out of training and stay visible in Claude by blocking only ClaudeBot.
The three Anthropic bots at a glance
| Bot | What it does | Follows robots.txt? | If you block it |
|---|---|---|---|
| ClaudeBot | Collects public web content that may contribute to training Anthropic's generative AI models. | Yes | Signals that your site's future materials should be excluded from Anthropic's training datasets. |
| Claude-SearchBot | Indexes web content to improve the relevance and accuracy of Claude's search responses. | Yes | Your content isn't indexed for search, which may reduce your visibility and accuracy in Claude's search results. |
| Claude-User | Retrieves a page when a Claude user asks a question that requires it. | Yes | Claude can't retrieve your content in response to a user's query, which may reduce your visibility for user-directed web search. |
All three share one published IP list, claude.com/crawling/bots.json.
ClaudeBot: model training
ClaudeBot is the training crawler. Anthropic says restricting it signals that your site's future materials should be excluded from its AI model training datasets. It has no role in Claude's search index or in user fetches, so blocking it doesn't stop Claude from finding or reading your pages for users.
Claude-SearchBot: Claude search
Claude-SearchBot builds the index Claude uses to answer with search. Anthropic says disabling it prevents your content from being indexed for search, which may reduce your site's visibility and accuracy in Claude's search results. If you want Claude to cite you, allow it.
Claude-User: user-triggered fetches
Claude-User visits your page when someone asks Claude a question that needs it. Here Anthropic differs from OpenAI and Perplexity: it says its bots honor robots.txt, and lists Claude-User with its own opt-out. A Disallow for Claude-User does stop Claude from retrieving your pages for users. OpenAI, by contrast, says robots.txt may not apply to ChatGPT-User (see GPTBot vs OAI-SearchBot vs ChatGPT-User).
What blocking each one changes
| If you disallow | Training | Claude search | User-triggered fetches |
|---|---|---|---|
| ClaudeBot only | Future content excluded | Unchanged | Unchanged |
| Claude-SearchBot only | Unchanged | Not indexed; visibility may drop | Unchanged |
| Claude-User only | Unchanged | Unchanged | Claude can't fetch your pages for users |
| All three | Future content excluded | Not indexed | Claude can't fetch your pages for users |
robots.txt recipes
Stay visible in Claude, opt out of training:
User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: /
Slow ClaudeBot down instead of blocking it. Anthropic supports the non-standard Crawl-delay directive:
User-agent: ClaudeBot Crawl-delay: 1
Opt out of Anthropic's bots entirely, including search and user fetches:
User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: /
Put the rules in the robots.txt at your site root and repeat them on every subdomain you want to opt out, as Anthropic instructs: docs.yourdomain.com reads its own robots.txt, not yours. For the full file across vendors, see robots.txt for AI crawlers, or pick bots one by one in the AI robots.txt generator.
How to verify Anthropic's bots
Anthropic publishes one IP list for its crawlers at claude.com/crawling/bots.json and says that if a request's source IP is on that list, it is coming from Anthropic. Because the list is shared, the IP tells you the request is from Anthropic, and the bot name in the user agent tells you which of the three it claims to be. To list the current ranges:
curl -s https://claude.com/crawling/bots.json | jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix'
Common mistakes
- Blocking Anthropic's IP addresses to opt out of training. Anthropic says this may not work reliably, because it stops the bots from reading your robots.txt. Use robots.txt.
- Adding the rule only to the main domain. Anthropic says to add it to every subdomain you want to opt out.
- Blocking ClaudeBot and expecting to disappear from Claude. Claude-SearchBot and Claude-User have separate opt-outs.
- Disallowing Claude-User on the assumption that user-triggered bots ignore robots.txt anyway. Anthropic says its bots honor it, so the rule takes effect and Claude can't read your pages for users.
- Putting a CAPTCHA or bot challenge in front of pages you want Claude to read. Anthropic says its bots respect anti-circumvention technologies and don't try to bypass CAPTCHAs.
Check what Anthropic's bots actually get
The AI crawler checker requests your site as Claude-SearchBot, Claude-User, ClaudeBot and other AI bots, so it shows CDN and firewall blocks that robots.txt alone doesn't. The free scan below checks your robots.txt rules for AI search bots, plus readable content, structured data and llms.txt.