# ClaudeBot

> Collects public web content that may contribute to training Anthropic's generative AI models. Restricting it tells Anthropic to exclude your future content from training datasets.

Canonical page: https://getrankroot.com/ai-bots/claudebot · Facts checked against Anthropic docs on 2026-10-20

| Operator | Used for | Type | Obeys robots.txt |
| --- | --- | --- | --- |
| Anthropic | Anthropic model training | AI training crawler | Yes |

## Should you allow it?

Your choice. Blocking it does not affect Claude search (Claude-SearchBot) or user fetches (Claude-User).

**If you block it:** Signals Anthropic to exclude your future content from its training datasets. Claude search and user fetches are unaffected.

Anthropic warns that IP blocking may not reliably opt you out; use robots.txt on every subdomain.

## robots.txt rules

Allow ClaudeBot:

```
User-agent: ClaudeBot
Allow: /
```

Block ClaudeBot:

```
User-agent: ClaudeBot
Disallow: /
```

Disallow ClaudeBot from one folder only:

```
User-agent: ClaudeBot
Disallow: /private/
Allow: /
```

A bot follows the most specific User-agent group that names it and ignores the generic `User-agent: *` group, so repeat any shared Disallow lines inside it.

## Same company, different bots: Anthropic

| Bot | Type | Purpose | Obeys robots.txt | If you block it |
| --- | --- | --- | --- | --- |
| [Claude-SearchBot](https://getrankroot.com/ai-bots/claude-searchbot.md) | AI search index | Claude search | Yes | Your content is not indexed for Claude search, which may reduce your visibility in Claude's answers. |
| [Claude-User](https://getrankroot.com/ai-bots/claude-user.md) | User-triggered fetcher | Claude (user requests) | Yes | Claude cannot retrieve your pages in response to users' questions. |
| [ClaudeBot](https://getrankroot.com/ai-bots/claudebot.md) | AI training crawler | Anthropic model training | Yes | Signals Anthropic to exclude your future content from its training datasets. Claude search and user fetches are unaffected. |

## How to verify it's really ClaudeBot

Find its requests in your server logs:

```
grep -i "ClaudeBot" access.log
```

Anthropic publishes one IP list for its crawlers. If a request's source IP is on that list, Anthropic says it is coming from Anthropic.

Published IP list: https://claude.com/crawling/bots.json

## Common mistakes

- Blocking Anthropic's IP addresses instead of using robots.txt. Anthropic says this may not reliably opt you out because it stops the bot from reading your robots.txt.
- Adding the rule only to the main domain. Anthropic says to add it to the robots.txt of every subdomain you want to opt out.
- Writing a User-agent: ClaudeBot group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the ClaudeBot group.
- Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
- Trusting the user agent header alone. Anyone can send "ClaudeBot" in a request; verify the source IP as shown above before allowlisting or blocking.

## FAQ

### What is ClaudeBot?

Collects public web content that may contribute to training Anthropic's generative AI models. Restricting it tells Anthropic to exclude your future content from training datasets.

### Does ClaudeBot respect robots.txt?

Yes, according to Anthropic.

### What happens if I block ClaudeBot?

Signals Anthropic to exclude your future content from its training datasets. Claude search and user fetches are unaffected.

### Should I block ClaudeBot?

Your choice. Blocking it does not affect Claude search (Claude-SearchBot) or user fetches (Claude-User).

### How do I block ClaudeBot?

Add User-agent: ClaudeBot and Disallow: / to the robots.txt in your top-level directory, and repeat it on every subdomain you want to opt out. Anthropic says robots.txt is the way to opt out.

### Can I slow ClaudeBot down instead of blocking it?

Yes. Anthropic supports the non-standard Crawl-delay directive, for example Crawl-delay: 1 in the ClaudeBot group.

### Does ClaudeBot bypass CAPTCHAs or logins?

Anthropic says its bots respect anti-circumvention technologies and do not attempt to bypass CAPTCHAs.

## Sources

- [Official documentation from Anthropic](https://support.claude.com/en/articles/8896518-what-is-claudebot)
- [Published IP addresses for ClaudeBot](https://claude.com/crawling/bots.json)

Check whether ClaudeBot can reach a site: https://getrankroot.com/tools/ai-crawler-checker or `GET https://getrankroot.com/api/bots?url=example.com`. Full list: https://getrankroot.com/ai-bots.md