AI crawler list · Anthropic · AI training crawler

ClaudeBot

Collects public web content that may contribute to training Anthropic's generative AI models. Restricting it tells Anthropic to exclude your future content from training datasets.

Facts checked against Anthropic docs on Oct 20, 2026.

Operator

Anthropic

Used for

Anthropic model training

Type

AI training crawler

Obeys robots.txt

Yes

Is ClaudeBot allowed on your site?

Reads your robots.txt and requests your page as each AI bot. Free, no signup.

Should you allow it?

Your choice. Blocking it does not affect Claude search (Claude-SearchBot) or user fetches (Claude-User).

If you block it: Signals Anthropic to exclude your future content from its training datasets. Claude search and user fetches are unaffected.

Anthropic warns that IP blocking may not reliably opt you out; use robots.txt on every subdomain.

robots.txt rules

Allow ClaudeBot

User-agent: ClaudeBot
Allow: /

Block ClaudeBot

User-agent: ClaudeBot
Disallow: /

Disallow ClaudeBot from one folder only

User-agent: ClaudeBot
Disallow: /private/
Allow: /

Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.

Same company, different bots: Anthropic

Anthropic uses separate user agents for separate jobs. Blocking one does not block the others.

Bot	Purpose	Obeys robots.txt	If you block it
`Claude-SearchBot` AI search index	Claude search	Yes	Your content is not indexed for Claude search, which may reduce your visibility in Claude's answers.
`Claude-User` User-triggered fetcher	Claude (user requests)	Yes	Claude cannot retrieve your pages in response to users' questions.
`ClaudeBot` AI training crawler	Anthropic model training	Yes	Signals Anthropic to exclude your future content from its training datasets. Claude search and user fetches are unaffected.

Allow Anthropic search and user fetches, opt out of Anthropic training

User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

How to verify it's really ClaudeBot

Find its requests in your server logs (the path to your access log depends on your host):

grep -i "ClaudeBot" access.log

Anthropic's documentation does not publish the full user agent string for ClaudeBot, so treat a token match as a starting point and confirm the IP.

Anthropic publishes one IP list for its crawlers. If a request's source IP is on that list, Anthropic says it is coming from Anthropic.

Published IP list: claude.com/crawling/bots.json

Check one IP against the list (prints True or False):

curl -s https://claude.com/crawling/bots.json | python3 -c "import sys,re,ipaddress as a; ip=a.ip_address(sys.argv[1]); print(any(ip in a.ip_network(c,strict=False) for c in re.findall(r'[0-9a-fA-F:.]+/[0-9]+', sys.stdin.read())))" 203.0.113.7

Common mistakes

Blocking Anthropic's IP addresses instead of using robots.txt. Anthropic says this may not reliably opt you out because it stops the bot from reading your robots.txt.
Adding the rule only to the main domain. Anthropic says to add it to the robots.txt of every subdomain you want to opt out.
Writing a User-agent: ClaudeBot group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the ClaudeBot group.
Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
Trusting the user agent header alone. Anyone can send "ClaudeBot" in a request; verify the source IP as shown above before allowlisting or blocking.

FAQ

What is ClaudeBot?

Collects public web content that may contribute to training Anthropic's generative AI models. Restricting it tells Anthropic to exclude your future content from training datasets.

Does ClaudeBot respect robots.txt?

Yes, according to Anthropic.

What happens if I block ClaudeBot?

Signals Anthropic to exclude your future content from its training datasets. Claude search and user fetches are unaffected.

Should I block ClaudeBot?

Your choice. Blocking it does not affect Claude search (Claude-SearchBot) or user fetches (Claude-User).

How do I block ClaudeBot?

Add User-agent: ClaudeBot and Disallow: / to the robots.txt in your top-level directory, and repeat it on every subdomain you want to opt out. Anthropic says robots.txt is the way to opt out.

Can I slow ClaudeBot down instead of blocking it?

Yes. Anthropic supports the non-standard Crawl-delay directive, for example Crawl-delay: 1 in the ClaudeBot group.

Does ClaudeBot bypass CAPTCHAs or logins?

Anthropic says its bots respect anti-circumvention technologies and do not attempt to bypass CAPTCHAs.

Sources

Related AI crawlers

GPTBot · Claude-SearchBot · Claude-User · meta-externalagent · Amazonbot · MistralAI-Training · CCBot

Facts checked against Anthropic docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me