AI crawler list · Anthropic · User-triggered fetcher

Claude-User

Fetches a page when a Claude user asks a question that needs it. Anthropic says disabling it prevents Claude from retrieving your content in response to a user's query, which may reduce your visibility for user-directed web search.

Facts checked against Anthropic docs on Oct 20, 2026.

Operator

Anthropic

Used for

Claude (user requests)

Type

User-triggered fetcher

Obeys robots.txt

Yes

Is Claude-User allowed on your site?

Reads your robots.txt and requests your page as each AI bot. Free, no signup.

Should you allow it?

Allow it. Blocking it means Claude cannot read your page even when a user asks about it.

If you block it: Claude cannot retrieve your pages in response to users' questions.

Unlike ChatGPT-User and Perplexity-User, Anthropic says its bots honor robots.txt, so blocking Claude-User does keep Claude from reading your pages for users.

robots.txt rules

Allow Claude-User

User-agent: Claude-User
Allow: /

Block Claude-User

User-agent: Claude-User
Disallow: /

Disallow Claude-User from one folder only

User-agent: Claude-User
Disallow: /private/
Allow: /

Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.

Same company, different bots: Anthropic

Anthropic uses separate user agents for separate jobs. Blocking one does not block the others.

Bot	Purpose	Obeys robots.txt	If you block it
`Claude-SearchBot` AI search index	Claude search	Yes	Your content is not indexed for Claude search, which may reduce your visibility in Claude's answers.
`Claude-User` User-triggered fetcher	Claude (user requests)	Yes	Claude cannot retrieve your pages in response to users' questions.
`ClaudeBot` AI training crawler	Anthropic model training	Yes	Signals Anthropic to exclude your future content from its training datasets. Claude search and user fetches are unaffected.

Allow Anthropic search and user fetches, opt out of Anthropic training

User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

How to verify it's really Claude-User

Find its requests in your server logs (the path to your access log depends on your host):

grep -i "Claude-User" access.log

Anthropic's documentation does not publish the full user agent string for Claude-User, so treat a token match as a starting point and confirm the IP.

Anthropic publishes one IP list for its crawlers. If a request's source IP is on that list, Anthropic says it is coming from Anthropic.

Published IP list: claude.com/crawling/bots.json

Check one IP against the list (prints True or False):

curl -s https://claude.com/crawling/bots.json | python3 -c "import sys,re,ipaddress as a; ip=a.ip_address(sys.argv[1]); print(any(ip in a.ip_network(c,strict=False) for c in re.findall(r'[0-9a-fA-F:.]+/[0-9]+', sys.stdin.read())))" 203.0.113.7

Common mistakes

Assuming a robots.txt rule won't matter because it is user-triggered. Anthropic says its bots honor robots.txt, so a Disallow for Claude-User does stop these fetches.
Writing a User-agent: Claude-User group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the Claude-User group.
Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
Trusting the user agent header alone. Anyone can send "Claude-User" in a request; verify the source IP as shown above before allowlisting or blocking.

FAQ

What is Claude-User?

Does Claude-User respect robots.txt?

Yes, according to Anthropic.

What happens if I block Claude-User?

Claude cannot retrieve your pages in response to users' questions.

Should I block Claude-User?

Allow it. Blocking it means Claude cannot read your page even when a user asks about it.

Does Claude-User respect robots.txt?

Yes. Anthropic says its bots honor industry-standard robots.txt directives, including Claude-User. That is different from OpenAI and Perplexity, which say their user-triggered fetchers may not follow robots.txt.

Does blocking ClaudeBot also block Claude-User?

No. Anthropic lists ClaudeBot (training), Claude-User (user requests) and Claude-SearchBot (search) as separate bots with separate opt-outs. Give each its own User-agent group.

How do I verify Claude-User traffic?

Anthropic publishes its crawler IP addresses at claude.com/crawling/bots.json. If the source IP is on that list, the request is from Anthropic.

Sources

Related AI crawlers

ChatGPT-User · OAI-AdsBot · Claude-SearchBot · ClaudeBot · Perplexity-User · Google-CloudVertexBot · meta-externalfetcher · Amzn-User

Facts checked against Anthropic docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me