# Claude-SearchBot

> Crawls the web to improve the quality of Claude's search results. Anthropic says disabling it prevents your content from being indexed for search, which may reduce your visibility and accuracy in Claude's search results.

Canonical page: https://getrankroot.com/ai-bots/claude-searchbot · Facts checked against Anthropic docs on 2026-10-20

| Operator | Used for | Type | Obeys robots.txt |
| --- | --- | --- | --- |
| Anthropic | Claude search | AI search index | Yes |

## Should you allow it?

Allow it if you want to be found and cited in Claude.

**If you block it:** Your content is not indexed for Claude search, which may reduce your visibility in Claude's answers.

Anthropic supports the non-standard Crawl-delay directive.

## robots.txt rules

Allow Claude-SearchBot:

```
User-agent: Claude-SearchBot
Allow: /
```

Block Claude-SearchBot:

```
User-agent: Claude-SearchBot
Disallow: /
```

Disallow Claude-SearchBot from one folder only:

```
User-agent: Claude-SearchBot
Disallow: /private/
Allow: /
```

A bot follows the most specific User-agent group that names it and ignores the generic `User-agent: *` group, so repeat any shared Disallow lines inside it.

## Same company, different bots: Anthropic

| Bot | Type | Purpose | Obeys robots.txt | If you block it |
| --- | --- | --- | --- | --- |
| [Claude-SearchBot](https://getrankroot.com/ai-bots/claude-searchbot.md) | AI search index | Claude search | Yes | Your content is not indexed for Claude search, which may reduce your visibility in Claude's answers. |
| [Claude-User](https://getrankroot.com/ai-bots/claude-user.md) | User-triggered fetcher | Claude (user requests) | Yes | Claude cannot retrieve your pages in response to users' questions. |
| [ClaudeBot](https://getrankroot.com/ai-bots/claudebot.md) | AI training crawler | Anthropic model training | Yes | Signals Anthropic to exclude your future content from its training datasets. Claude search and user fetches are unaffected. |

## How to verify it's really Claude-SearchBot

Find its requests in your server logs:

```
grep -i "Claude-SearchBot" access.log
```

Anthropic publishes one IP list for its crawlers. If a request's source IP is on that list, Anthropic says it is coming from Anthropic.

Published IP list: https://claude.com/crawling/bots.json

## Common mistakes

- Writing a User-agent: Claude-SearchBot group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the Claude-SearchBot group.
- Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
- Trusting the user agent header alone. Anyone can send "Claude-SearchBot" in a request; verify the source IP as shown above before allowlisting or blocking.

## FAQ

### What is Claude-SearchBot?

Crawls the web to improve the quality of Claude's search results. Anthropic says disabling it prevents your content from being indexed for search, which may reduce your visibility and accuracy in Claude's search results.

### Does Claude-SearchBot respect robots.txt?

Yes, according to Anthropic.

### What happens if I block Claude-SearchBot?

Your content is not indexed for Claude search, which may reduce your visibility in Claude's answers.

### Should I block Claude-SearchBot?

Allow it if you want to be found and cited in Claude.

## Sources

- [Official documentation from Anthropic](https://support.claude.com/en/articles/8896518-what-is-claudebot)
- [Published IP addresses for Claude-SearchBot](https://claude.com/crawling/bots.json)

Check whether Claude-SearchBot can reach a site: https://getrankroot.com/tools/ai-crawler-checker or `GET https://getrankroot.com/api/bots?url=example.com`. Full list: https://getrankroot.com/ai-bots.md