# meta-externalagent

> Crawls for training foundation AI models or indexing content directly to improve Meta products.

Canonical page: https://getrankroot.com/ai-bots/meta-externalagent · Facts checked against Meta docs on 2026-10-20

| Operator | Used for | Type | Obeys robots.txt |
| --- | --- | --- | --- |
| Meta | Meta AI training and indexing | AI training crawler | Yes |

## Should you allow it?

Your choice. Meta's search-citation crawler is meta-webindexer.

**If you block it:** Stops the crawler Meta uses to gather content for training its AI models and for product indexing.

Meta says robots.txt changes can take up to 24 hours because its crawlers may cache the file.

Meta lists the user agent as meta-externalagent/1.1, sometimes followed by a link to its crawler docs, so match on the token.

Meta documents a separate meta-externalads crawler for advertising and other business products; disallow it on its own if you want it blocked too.

## robots.txt rules

Allow meta-externalagent:

```
User-agent: meta-externalagent
Allow: /
```

Block meta-externalagent:

```
User-agent: meta-externalagent
Disallow: /
```

Disallow meta-externalagent from one folder only:

```
User-agent: meta-externalagent
Disallow: /private/
Allow: /
```

A bot follows the most specific User-agent group that names it and ignores the generic `User-agent: *` group, so repeat any shared Disallow lines inside it.

## Same company, different bots: Meta

| Bot | Type | Purpose | Obeys robots.txt | If you block it |
| --- | --- | --- | --- | --- |
| [meta-externalagent](https://getrankroot.com/ai-bots/meta-externalagent.md) | AI training crawler | Meta AI training and indexing | Yes | Stops the crawler Meta uses to gather content for training its AI models and for product indexing. |
| [meta-webindexer](https://getrankroot.com/ai-bots/meta-webindexer.md) | AI search index | Meta AI search | Yes | Meta AI is less able to cite and link to your content in its answers. |
| [meta-externalfetcher](https://getrankroot.com/ai-bots/meta-externalfetcher.md) | User-triggered fetcher | Meta AI (user requests) | Not reliably | A robots.txt rule may have little effect; Meta says it may bypass robots.txt. |

## User agent string

```
meta-externalagent/1.1
```

## How to verify it's really meta-externalagent

Find its requests in your server logs:

```
grep -i "meta-externalagent" access.log
```

Meta does not publish an IP list or reverse DNS method for meta-externalagent, so the user agent is the only signal.

## Common mistakes

- Writing a User-agent: meta-externalagent group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the meta-externalagent group.
- Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.

## FAQ

### What is meta-externalagent?

Crawls for training foundation AI models or indexing content directly to improve Meta products.

### Does meta-externalagent respect robots.txt?

Yes, according to Meta.

### What happens if I block meta-externalagent?

Stops the crawler Meta uses to gather content for training its AI models and for product indexing.

### Should I block meta-externalagent?

Your choice. Meta's search-citation crawler is meta-webindexer.

## Sources

- [Official documentation from Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)

Check whether meta-externalagent can reach a site: https://getrankroot.com/tools/ai-crawler-checker or `GET https://getrankroot.com/api/bots?url=example.com`. Full list: https://getrankroot.com/ai-bots.md