# CCBot

> Common Crawl's crawler. It builds a free, open web archive that many AI labs and researchers use as training data.

Canonical page: https://getrankroot.com/ai-bots/ccbot · Facts checked against Common Crawl docs on 2026-10-20

| Operator | Used for | Type | Obeys robots.txt |
| --- | --- | --- | --- |
| Common Crawl | Common Crawl open dataset | AI training crawler | Yes |

## Should you allow it?

Your choice. Blocking it keeps your future pages out of a dataset widely used for training.

**If you block it:** Common Crawl stops crawling your site, keeping future pages out of its open dataset.

Common Crawl warns that other crawlers sometimes pretend to be CCBot; verify with reverse DNS.

## robots.txt rules

Allow CCBot:

```
User-agent: CCBot
Allow: /
```

Block CCBot:

```
User-agent: CCBot
Disallow: /
```

Disallow CCBot from one folder only:

```
User-agent: CCBot
Disallow: /private/
Allow: /
```

A bot follows the most specific User-agent group that names it and ignores the generic `User-agent: *` group, so repeat any shared Disallow lines inside it.

## User agent string

```
CCBot/2.0 (https://commoncrawl.org/faq/)
```

## How to verify it's really CCBot

Find its requests in your server logs:

```
grep -i "CCBot" access.log
```

Common Crawl warns that other crawlers sometimes pretend to be CCBot. Its IPv6 ranges have no reverse DNS, so check those against the IP list.

Reverse DNS check (Common Crawl documents this): the hostname of the IP must end in `crawl.commoncrawl.org`, and a forward lookup of that hostname must return the same IP.

Published IP list: https://index.commoncrawl.org/ccbot.json

## Common mistakes

- Writing a User-agent: CCBot group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the CCBot group.
- Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
- Trusting the user agent header alone. Anyone can send "CCBot" in a request; verify the source IP as shown above before allowlisting or blocking.

## FAQ

### What is CCBot?

Common Crawl's crawler. It builds a free, open web archive that many AI labs and researchers use as training data.

### Does CCBot respect robots.txt?

Yes, according to Common Crawl.

### What happens if I block CCBot?

Common Crawl stops crawling your site, keeping future pages out of its open dataset.

### Should I block CCBot?

Your choice. Blocking it keeps your future pages out of a dataset widely used for training.

## Sources

- [Official documentation from Common Crawl](https://commoncrawl.org/ccbot)
- [Published IP addresses for CCBot](https://index.commoncrawl.org/ccbot.json)

Check whether CCBot can reach a site: https://getrankroot.com/tools/ai-crawler-checker or `GET https://getrankroot.com/api/bots?url=example.com`. Full list: https://getrankroot.com/ai-bots.md