# meta-externalfetcher

> Fetches individual links at a user's request, including for agentic AI tasks. Meta says it may bypass robots.txt because the fetch is user-requested.

Canonical page: https://getrankroot.com/ai-bots/meta-externalfetcher · Facts checked against Meta docs on 2026-10-20

| Operator | Used for | Type | Obeys robots.txt |
| --- | --- | --- | --- |
| Meta | Meta AI (user requests) | User-triggered fetcher | Not reliably |

## Should you allow it?

Leave it allowed.

**If you block it:** A robots.txt rule may have little effect; Meta says it may bypass robots.txt.

## robots.txt rules

Allow meta-externalfetcher:

```
User-agent: meta-externalfetcher
Allow: /
```

Block meta-externalfetcher:

```
User-agent: meta-externalfetcher
Disallow: /
```

Disallow meta-externalfetcher from one folder only:

```
User-agent: meta-externalfetcher
Disallow: /private/
Allow: /
```

A bot follows the most specific User-agent group that names it and ignores the generic `User-agent: *` group, so repeat any shared Disallow lines inside it.

This fetcher does not reliably follow robots.txt, so a Disallow rule may have little effect.

## Same company, different bots: Meta

| Bot | Type | Purpose | Obeys robots.txt | If you block it |
| --- | --- | --- | --- | --- |
| [meta-externalagent](https://getrankroot.com/ai-bots/meta-externalagent.md) | AI training crawler | Meta AI training and indexing | Yes | Stops the crawler Meta uses to gather content for training its AI models and for product indexing. |
| [meta-webindexer](https://getrankroot.com/ai-bots/meta-webindexer.md) | AI search index | Meta AI search | Yes | Meta AI is less able to cite and link to your content in its answers. |
| [meta-externalfetcher](https://getrankroot.com/ai-bots/meta-externalfetcher.md) | User-triggered fetcher | Meta AI (user requests) | Not reliably | A robots.txt rule may have little effect; Meta says it may bypass robots.txt. |

## User agent string

```
meta-externalfetcher/1.1
```

## How to verify it's really meta-externalfetcher

Find its requests in your server logs:

```
grep -i "meta-externalfetcher" access.log
```

Meta does not publish an IP list or reverse DNS method for meta-externalfetcher, so the user agent is the only signal.

## Common mistakes

- Writing a User-agent: meta-externalfetcher group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the meta-externalfetcher group.
- Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.

## FAQ

### What is meta-externalfetcher?

Fetches individual links at a user's request, including for agentic AI tasks. Meta says it may bypass robots.txt because the fetch is user-requested.

### Does meta-externalfetcher respect robots.txt?

Not reliably. Meta says user-requested fetches may not follow robots.txt.

### What happens if I block meta-externalfetcher?

A robots.txt rule may have little effect; Meta says it may bypass robots.txt.

### Should I block meta-externalfetcher?

Leave it allowed.

## Sources

- [Official documentation from Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)

Check whether meta-externalfetcher can reach a site: https://getrankroot.com/tools/ai-crawler-checker or `GET https://getrankroot.com/api/bots?url=example.com`. Full list: https://getrankroot.com/ai-bots.md