AI crawler list · Common Crawl · AI training crawler
CCBot
Common Crawl's crawler. It builds a free, open web archive that many AI labs and researchers use as training data.
Facts checked against Common Crawl docs on Oct 20, 2026.
Is CCBot allowed on your site?
Reads your robots.txt and requests your page as each AI bot. Free, no signup.
Should you allow it?
Your choice. Blocking it keeps your future pages out of a dataset widely used for training.
If you block it: Common Crawl stops crawling your site, keeping future pages out of its open dataset.
Common Crawl warns that other crawlers sometimes pretend to be CCBot; verify with reverse DNS.
robots.txt rules
User-agent: CCBot Allow: /
User-agent: CCBot Disallow: /
User-agent: CCBot Disallow: /private/ Allow: /
Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.
User agent string
CCBot/2.0 (https://commoncrawl.org/faq/)
Versions in the string can change. Match on the token CCBot, and verify real traffic as shown below rather than trusting the header.
How to verify it's really CCBot
Find its requests in your server logs (the path to your access log depends on your host):
grep -i "CCBot" access.log
Common Crawl warns that other crawlers sometimes pretend to be CCBot. Its IPv6 ranges have no reverse DNS, so check those against the IP list.
Reverse DNS check (Common Crawl documents this): look up the IP from your log, confirm the hostname ends in crawl.commoncrawl.org, then look up that hostname and confirm it returns the same IP.
host 203.0.113.7 # replace with the IP from your log host <hostname-from-above> # must return the same IP
Published IP list: index.commoncrawl.org/ccbot.json
Check one IP against the list (prints True or False):
curl -s https://index.commoncrawl.org/ccbot.json | python3 -c "import sys,re,ipaddress as a; ip=a.ip_address(sys.argv[1]); print(any(ip in a.ip_network(c,strict=False) for c in re.findall(r'[0-9a-fA-F:.]+/[0-9]+', sys.stdin.read())))" 203.0.113.7
Common mistakes
- Writing a User-agent: CCBot group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the CCBot group.
- Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
- Trusting the user agent header alone. Anyone can send "CCBot" in a request; verify the source IP as shown above before allowlisting or blocking.
FAQ
Common Crawl's crawler. It builds a free, open web archive that many AI labs and researchers use as training data.
Yes, according to Common Crawl.
Common Crawl stops crawling your site, keeping future pages out of its open dataset.
Your choice. Blocking it keeps your future pages out of a dataset widely used for training.
Sources
- Official documentation from Common Crawl
- Published IP addresses for CCBot
- Cloudflare bots changelog (AI bot policy defaults)
Related AI crawlers
GPTBot · ClaudeBot · meta-externalagent · Amazonbot · MistralAI-Training
Facts checked against Common Crawl docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me