AI crawler list · Meta · User-triggered fetcher
meta-externalfetcher
Fetches individual links at a user's request, including for agentic AI tasks. Meta says it may bypass robots.txt because the fetch is user-requested.
Facts checked against Meta docs on Oct 20, 2026.
Is meta-externalfetcher allowed on your site?
Reads your robots.txt and requests your page as each AI bot. Free, no signup.
Should you allow it?
Leave it allowed.
If you block it: A robots.txt rule may have little effect; Meta says it may bypass robots.txt.
robots.txt rules
User-agent: meta-externalfetcher Allow: /
User-agent: meta-externalfetcher Disallow: /
User-agent: meta-externalfetcher Disallow: /private/ Allow: /
Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.
This fetcher does not reliably follow robots.txt, so a Disallow rule may have little effect.
Same company, different bots: Meta
Meta uses separate user agents for separate jobs. Blocking one does not block the others.
| Bot | Purpose | Obeys robots.txt | If you block it |
|---|---|---|---|
meta-externalagentAI training crawler | Meta AI training and indexing | Yes | Stops the crawler Meta uses to gather content for training its AI models and for product indexing. |
meta-webindexerAI search index | Meta AI search | Yes | Meta AI is less able to cite and link to your content in its answers. |
meta-externalfetcherUser-triggered fetcher | Meta AI (user requests) | Not reliably | A robots.txt rule may have little effect; Meta says it may bypass robots.txt. |
User-agent: meta-webindexer User-agent: meta-externalfetcher Allow: / User-agent: meta-externalagent Disallow: /
User agent string
meta-externalfetcher/1.1
Versions in the string can change. Match on the token meta-externalfetcher, and verify real traffic as shown below rather than trusting the header.
How to verify it's really meta-externalfetcher
Find its requests in your server logs (the path to your access log depends on your host):
grep -i "meta-externalfetcher" access.log
Meta does not publish an IP list or reverse DNS method for meta-externalfetcher, so the user agent is the only signal.
Common mistakes
- Writing a User-agent: meta-externalfetcher group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the meta-externalfetcher group.
- Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
FAQ
Fetches individual links at a user's request, including for agentic AI tasks. Meta says it may bypass robots.txt because the fetch is user-requested.
Not reliably. Meta says user-requested fetches may not follow robots.txt.
A robots.txt rule may have little effect; Meta says it may bypass robots.txt.
Leave it allowed.
Sources
Related AI crawlers
ChatGPT-User · OAI-AdsBot · Claude-User · Perplexity-User · Google-CloudVertexBot · meta-externalagent · meta-webindexer · Amzn-User
Facts checked against Meta docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me