AI crawler list · Anthropic · AI search index

Claude-SearchBot

Crawls the web to improve the quality of Claude's search results. Anthropic says disabling it prevents your content from being indexed for search, which may reduce your visibility and accuracy in Claude's search results.

Facts checked against Anthropic docs on Oct 20, 2026.

Operator

Anthropic

Used for

Claude search

Type

AI search index

Obeys robots.txt

Yes

Is Claude-SearchBot allowed on your site?

Reads your robots.txt and requests your page as each AI bot. Free, no signup.

Should you allow it?

Allow it if you want to be found and cited in Claude.

If you block it: Your content is not indexed for Claude search, which may reduce your visibility in Claude's answers.

Anthropic supports the non-standard Crawl-delay directive.

robots.txt rules

Allow Claude-SearchBot

User-agent: Claude-SearchBot
Allow: /

Block Claude-SearchBot

User-agent: Claude-SearchBot
Disallow: /

Disallow Claude-SearchBot from one folder only

User-agent: Claude-SearchBot
Disallow: /private/
Allow: /

Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.

Same company, different bots: Anthropic

Anthropic uses separate user agents for separate jobs. Blocking one does not block the others.

Bot	Purpose	Obeys robots.txt	If you block it
`Claude-SearchBot` AI search index	Claude search	Yes	Your content is not indexed for Claude search, which may reduce your visibility in Claude's answers.
`Claude-User` User-triggered fetcher	Claude (user requests)	Yes	Claude cannot retrieve your pages in response to users' questions.
`ClaudeBot` AI training crawler	Anthropic model training	Yes	Signals Anthropic to exclude your future content from its training datasets. Claude search and user fetches are unaffected.

Allow Anthropic search and user fetches, opt out of Anthropic training

User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

How to verify it's really Claude-SearchBot

Find its requests in your server logs (the path to your access log depends on your host):

grep -i "Claude-SearchBot" access.log

Anthropic's documentation does not publish the full user agent string for Claude-SearchBot, so treat a token match as a starting point and confirm the IP.

Anthropic publishes one IP list for its crawlers. If a request's source IP is on that list, Anthropic says it is coming from Anthropic.

Published IP list: claude.com/crawling/bots.json

Check one IP against the list (prints True or False):

curl -s https://claude.com/crawling/bots.json | python3 -c "import sys,re,ipaddress as a; ip=a.ip_address(sys.argv[1]); print(any(ip in a.ip_network(c,strict=False) for c in re.findall(r'[0-9a-fA-F:.]+/[0-9]+', sys.stdin.read())))" 203.0.113.7

Common mistakes

Writing a User-agent: Claude-SearchBot group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the Claude-SearchBot group.
Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
Trusting the user agent header alone. Anyone can send "Claude-SearchBot" in a request; verify the source IP as shown above before allowlisting or blocking.

FAQ

What is Claude-SearchBot?

Does Claude-SearchBot respect robots.txt?

Yes, according to Anthropic.

What happens if I block Claude-SearchBot?

Your content is not indexed for Claude search, which may reduce your visibility in Claude's answers.

Should I block Claude-SearchBot?

Allow it if you want to be found and cited in Claude.

Sources

Related AI crawlers

OAI-SearchBot · Claude-User · ClaudeBot · PerplexityBot · Googlebot · Bingbot · Applebot · meta-webindexer

Facts checked against Anthropic docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me