# meta-webindexer

> Analyzes web content to improve Meta AI search results. Meta says allowing it helps Meta AI cite and link to your content.

Canonical page: https://getrankroot.com/ai-bots/meta-webindexer · Facts checked against Meta docs on 2026-10-20

| Operator | Used for | Type | Obeys robots.txt |
| --- | --- | --- | --- |
| Meta | Meta AI search | AI search index | Yes |

## Should you allow it?

Allow it if you want citations in Meta AI.

**If you block it:** Meta AI is less able to cite and link to your content in its answers.

## robots.txt rules

Allow meta-webindexer:

```
User-agent: meta-webindexer
Allow: /
```

Block meta-webindexer:

```
User-agent: meta-webindexer
Disallow: /
```

Disallow meta-webindexer from one folder only:

```
User-agent: meta-webindexer
Disallow: /private/
Allow: /
```

A bot follows the most specific User-agent group that names it and ignores the generic `User-agent: *` group, so repeat any shared Disallow lines inside it.

## Same company, different bots: Meta

| Bot | Type | Purpose | Obeys robots.txt | If you block it |
| --- | --- | --- | --- | --- |
| [meta-externalagent](https://getrankroot.com/ai-bots/meta-externalagent.md) | AI training crawler | Meta AI training and indexing | Yes | Stops the crawler Meta uses to gather content for training its AI models and for product indexing. |
| [meta-webindexer](https://getrankroot.com/ai-bots/meta-webindexer.md) | AI search index | Meta AI search | Yes | Meta AI is less able to cite and link to your content in its answers. |
| [meta-externalfetcher](https://getrankroot.com/ai-bots/meta-externalfetcher.md) | User-triggered fetcher | Meta AI (user requests) | Not reliably | A robots.txt rule may have little effect; Meta says it may bypass robots.txt. |

## User agent string

```
meta-webindexer/1.1
```

## How to verify it's really meta-webindexer

Find its requests in your server logs:

```
grep -i "meta-webindexer" access.log
```

Meta does not publish an IP list or reverse DNS method for meta-webindexer, so the user agent is the only signal.

## Common mistakes

- Writing a User-agent: meta-webindexer group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the meta-webindexer group.
- Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.

## FAQ

### What is meta-webindexer?

Analyzes web content to improve Meta AI search results. Meta says allowing it helps Meta AI cite and link to your content.

### Does meta-webindexer respect robots.txt?

Yes, according to Meta.

### What happens if I block meta-webindexer?

Meta AI is less able to cite and link to your content in its answers.

### Should I block meta-webindexer?

Allow it if you want citations in Meta AI.

## Sources

- [Official documentation from Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)

Check whether meta-webindexer can reach a site: https://getrankroot.com/tools/ai-crawler-checker or `GET https://getrankroot.com/api/bots?url=example.com`. Full list: https://getrankroot.com/ai-bots.md