AI crawler list · Meta · AI training crawler
meta-externalagent
Crawls for training foundation AI models or indexing content directly to improve Meta products.
Facts checked against Meta docs on Oct 20, 2026.
Is meta-externalagent allowed on your site?
Reads your robots.txt and requests your page as each AI bot. Free, no signup.
Should you allow it?
Your choice. Meta's search-citation crawler is meta-webindexer.
If you block it: Stops the crawler Meta uses to gather content for training its AI models and for product indexing.
Meta says robots.txt changes can take up to 24 hours because its crawlers may cache the file.
Meta lists the user agent as meta-externalagent/1.1, sometimes followed by a link to its crawler docs, so match on the token.
Meta documents a separate meta-externalads crawler for advertising and other business products; disallow it on its own if you want it blocked too.
robots.txt rules
User-agent: meta-externalagent Allow: /
User-agent: meta-externalagent Disallow: /
User-agent: meta-externalagent Disallow: /private/ Allow: /
Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.
Same company, different bots: Meta
Meta uses separate user agents for separate jobs. Blocking one does not block the others.
| Bot | Purpose | Obeys robots.txt | If you block it |
|---|---|---|---|
meta-externalagentAI training crawler | Meta AI training and indexing | Yes | Stops the crawler Meta uses to gather content for training its AI models and for product indexing. |
meta-webindexerAI search index | Meta AI search | Yes | Meta AI is less able to cite and link to your content in its answers. |
meta-externalfetcherUser-triggered fetcher | Meta AI (user requests) | Not reliably | A robots.txt rule may have little effect; Meta says it may bypass robots.txt. |
User-agent: meta-webindexer User-agent: meta-externalfetcher Allow: / User-agent: meta-externalagent Disallow: /
User agent string
meta-externalagent/1.1
Versions in the string can change. Match on the token meta-externalagent, and verify real traffic as shown below rather than trusting the header.
How to verify it's really meta-externalagent
Find its requests in your server logs (the path to your access log depends on your host):
grep -i "meta-externalagent" access.log
Meta does not publish an IP list or reverse DNS method for meta-externalagent, so the user agent is the only signal.
Common mistakes
- Writing a User-agent: meta-externalagent group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the meta-externalagent group.
- Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
FAQ
Crawls for training foundation AI models or indexing content directly to improve Meta products.
Yes, according to Meta.
Stops the crawler Meta uses to gather content for training its AI models and for product indexing.
Your choice. Meta's search-citation crawler is meta-webindexer.
Sources
Related AI crawlers
GPTBot · ClaudeBot · meta-webindexer · meta-externalfetcher · Amazonbot · MistralAI-Training · CCBot
Facts checked against Meta docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me