# ClaudeBot vs Claude-SearchBot vs Claude-User explained

> ClaudeBot collects training data, Claude-SearchBot indexes for Claude search, Claude-User fetches pages for users. What blocking each one does.

Canonical page: https://getrankroot.com/guides/claudebot-vs-claude-searchbot-vs-claude-user · Updated 2026-10-17

Anthropic runs three bots with separate jobs. [ClaudeBot](https://getrankroot.com/ai-bots/claudebot) collects public web content that may be used to train Anthropic's models. [Claude-SearchBot](https://getrankroot.com/ai-bots/claude-searchbot) indexes content for Claude's search responses. [Claude-User](https://getrankroot.com/ai-bots/claude-user) fetches a page when a Claude user asks a question that needs it. Anthropic says its bots honor robots.txt, so you can opt out of training and stay visible in Claude by blocking only ClaudeBot.

## The three Anthropic bots at a glance

| Bot | What it does | Follows robots.txt? | If you block it |
| --- | --- | --- | --- |
| [ClaudeBot](https://getrankroot.com/ai-bots/claudebot) | Collects public web content that may contribute to training Anthropic's generative AI models. | Yes | Signals that your site's future materials should be excluded from Anthropic's training datasets. |
| [Claude-SearchBot](https://getrankroot.com/ai-bots/claude-searchbot) | Indexes web content to improve the relevance and accuracy of Claude's search responses. | Yes | Your content isn't indexed for search, which may reduce your visibility and accuracy in Claude's search results. |
| [Claude-User](https://getrankroot.com/ai-bots/claude-user) | Retrieves a page when a Claude user asks a question that requires it. | Yes | Claude can't retrieve your content in response to a user's query, which may reduce your visibility for user-directed web search. |

All three share one published IP list, [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json).

## ClaudeBot: model training

ClaudeBot is the training crawler. Anthropic says restricting it signals that your site's future materials should be excluded from its AI model training datasets. It has no role in Claude's search index or in user fetches, so blocking it doesn't stop Claude from finding or reading your pages for users.

## Claude-SearchBot: Claude search

Claude-SearchBot builds the index Claude uses to answer with search. Anthropic says disabling it prevents your content from being indexed for search, which may reduce your site's visibility and accuracy in Claude's search results. If you want Claude to cite you, allow it.

## Claude-User: user-triggered fetches

Claude-User visits your page when someone asks Claude a question that needs it. Here Anthropic differs from OpenAI and Perplexity: it says its bots honor robots.txt, and lists Claude-User with its own opt-out. A Disallow for Claude-User does stop Claude from retrieving your pages for users. OpenAI, by contrast, says robots.txt may not apply to [ChatGPT-User](https://getrankroot.com/ai-bots/chatgpt-user) (see [GPTBot vs OAI-SearchBot vs ChatGPT-User](https://getrankroot.com/guides/gptbot-vs-oai-searchbot-vs-chatgpt-user)).

## What blocking each one changes

| If you disallow | Training | Claude search | User-triggered fetches |
| --- | --- | --- | --- |
| ClaudeBot only | Future content excluded | Unchanged | Unchanged |
| Claude-SearchBot only | Unchanged | Not indexed; visibility may drop | Unchanged |
| Claude-User only | Unchanged | Unchanged | Claude can't fetch your pages for users |
| All three | Future content excluded | Not indexed | Claude can't fetch your pages for users |

## robots.txt recipes

Stay visible in Claude, opt out of training:

```
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /
```

Slow ClaudeBot down instead of blocking it. Anthropic supports the non-standard Crawl-delay directive:

```
User-agent: ClaudeBot
Crawl-delay: 1
```

Opt out of Anthropic's bots entirely, including search and user fetches:

```
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /
```

Put the rules in the robots.txt at your site root and repeat them on every subdomain you want to opt out, as Anthropic instructs: docs.yourdomain.com reads its own robots.txt, not yours. For the full file across vendors, see [robots.txt for AI crawlers](https://getrankroot.com/guides/robots-txt-ai-crawlers), or pick bots one by one in the [AI robots.txt generator](https://getrankroot.com/tools/robots-txt-generator).

## How to verify Anthropic's bots

Anthropic publishes one IP list for its crawlers at [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json) and says that if a request's source IP is on that list, it is coming from Anthropic. Because the list is shared, the IP tells you the request is from Anthropic, and the bot name in the user agent tells you which of the three it claims to be. To list the current ranges:

```
curl -s https://claude.com/crawling/bots.json | jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix'
```

## Common mistakes

- Blocking Anthropic's IP addresses to opt out of training. Anthropic says this may not work reliably, because it stops the bots from reading your robots.txt. Use robots.txt.
- Adding the rule only to the main domain. Anthropic says to add it to every subdomain you want to opt out.
- Blocking ClaudeBot and expecting to disappear from Claude. Claude-SearchBot and Claude-User have separate opt-outs.
- Disallowing Claude-User on the assumption that user-triggered bots ignore robots.txt anyway. Anthropic says its bots honor it, so the rule takes effect and Claude can't read your pages for users.
- Putting a CAPTCHA or bot challenge in front of pages you want Claude to read. Anthropic says its bots respect anti-circumvention technologies and don't try to bypass CAPTCHAs.

## Check what Anthropic's bots actually get

The [AI crawler checker](https://getrankroot.com/tools/ai-crawler-checker) requests your site as Claude-SearchBot, Claude-User, ClaudeBot and other AI bots, so it shows CDN and firewall blocks that robots.txt alone doesn't. The free scan below checks your robots.txt rules for AI search bots, plus readable content, structured data and llms.txt.

## Sources

- [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (read 2026-10-17)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [OpenAI: Overview of OpenAI crawlers, for the ChatGPT-User comparison (read 2026-10-17)](https://developers.openai.com/api/docs/bots)

## FAQ

### What is the difference between ClaudeBot and Claude-SearchBot?

ClaudeBot collects content that may be used to train Anthropic's models. Claude-SearchBot indexes content for Claude's search responses. Blocking ClaudeBot opts your future content out of training; blocking Claude-SearchBot may reduce your visibility in Claude's search results.

### What is the Claude-User user agent?

Claude-User is the name Anthropic gives the bot that fetches pages when a Claude user asks a question that needs them. Use Claude-User as the User-agent line in robots.txt. To confirm a request really comes from Anthropic, check its source IP against claude.com/crawling/bots.json.

### Does Claude-User respect robots.txt?

Yes. Anthropic says its bots honor robots.txt directives. That differs from OpenAI, which says robots.txt may not apply to ChatGPT-User.

### Does blocking ClaudeBot remove my site from Claude?

No. ClaudeBot is for training. Claude's search index uses Claude-SearchBot, and live fetches for users use Claude-User. Each has its own robots.txt opt-out.

### Can I slow Anthropic's crawlers down instead of blocking them?

Yes. Anthropic supports the non-standard Crawl-delay directive, for example Crawl-delay: 1 in the ClaudeBot group.

## Check your site

Run the free audit at https://getrankroot.com/ or call `GET https://getrankroot.com/api/audit?url=example.com`.

## More guides

- [How to get your product recommended by ChatGPT, Perplexity and Claude](https://getrankroot.com/guides/how-to-get-recommended-by-chatgpt.md)
- [What is llms.txt? Format, examples and a free generator](https://getrankroot.com/guides/what-is-llms-txt.md)
- [robots.txt for AI crawlers: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot](https://getrankroot.com/guides/robots-txt-ai-crawlers.md)
- [Schema markup for AI search: the JSON-LD that helps AI describe you](https://getrankroot.com/guides/schema-markup-for-ai-search.md)
- [How to make your SaaS agent-ready (MCP, OpenAPI, llms.txt)](https://getrankroot.com/guides/make-your-saas-agent-ready.md)
- [How to track traffic from ChatGPT, Perplexity and Claude in GA4](https://getrankroot.com/guides/track-ai-traffic-ga4.md)
- [GEO vs SEO: what generative engine optimization actually changes](https://getrankroot.com/guides/geo-vs-seo.md)
- [How to check if ChatGPT mentions your brand (manual method + tools)](https://getrankroot.com/guides/check-if-chatgpt-mentions-your-brand.md)
- [llms.txt for WordPress: how to add it (manual, Yoast, plugins)](https://getrankroot.com/guides/llms-txt-wordpress.md)
- [Should you block GPTBot? AI training vs search crawlers explained](https://getrankroot.com/guides/should-you-block-gptbot.md)
- [Schema markup for SaaS: the JSON-LD worth adding for AI search](https://getrankroot.com/guides/schema-markup-for-saas.md)
- [GPTBot vs OAI-SearchBot vs ChatGPT-User: what to block](https://getrankroot.com/guides/gptbot-vs-oai-searchbot-vs-chatgpt-user.md)
- [What is an AI visibility score? (and how to check yours)](https://getrankroot.com/guides/what-is-an-ai-visibility-score.md)