AI crawler list · Meta · AI search index

meta-webindexer

Analyzes web content to improve Meta AI search results. Meta says allowing it helps Meta AI cite and link to your content.

Facts checked against Meta docs on Oct 20, 2026.

Operator

Meta

Used for

Meta AI search

Type

AI search index

Obeys robots.txt

Yes

Is meta-webindexer allowed on your site?

Reads your robots.txt and requests your page as each AI bot. Free, no signup.

Should you allow it?

Allow it if you want citations in Meta AI.

If you block it: Meta AI is less able to cite and link to your content in its answers.

robots.txt rules

Allow meta-webindexer

User-agent: meta-webindexer
Allow: /

Block meta-webindexer

User-agent: meta-webindexer
Disallow: /

Disallow meta-webindexer from one folder only

User-agent: meta-webindexer
Disallow: /private/
Allow: /

Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.

Same company, different bots: Meta

Meta uses separate user agents for separate jobs. Blocking one does not block the others.

Bot	Purpose	Obeys robots.txt	If you block it
`meta-externalagent` AI training crawler	Meta AI training and indexing	Yes	Stops the crawler Meta uses to gather content for training its AI models and for product indexing.
`meta-webindexer` AI search index	Meta AI search	Yes	Meta AI is less able to cite and link to your content in its answers.
`meta-externalfetcher` User-triggered fetcher	Meta AI (user requests)	Not reliably	A robots.txt rule may have little effect; Meta says it may bypass robots.txt.

Allow Meta search and user fetches, opt out of Meta training

User-agent: meta-webindexer
User-agent: meta-externalfetcher
Allow: /

User-agent: meta-externalagent
Disallow: /

User agent string

meta-webindexer/1.1

Versions in the string can change. Match on the token meta-webindexer, and verify real traffic as shown below rather than trusting the header.

How to verify it's really meta-webindexer

Find its requests in your server logs (the path to your access log depends on your host):

grep -i "meta-webindexer" access.log

Meta does not publish an IP list or reverse DNS method for meta-webindexer, so the user agent is the only signal.

Common mistakes

Writing a User-agent: meta-webindexer group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the meta-webindexer group.
Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.

FAQ

What is meta-webindexer?

Analyzes web content to improve Meta AI search results. Meta says allowing it helps Meta AI cite and link to your content.

Does meta-webindexer respect robots.txt?

Yes, according to Meta.

What happens if I block meta-webindexer?

Meta AI is less able to cite and link to your content in its answers.

Should I block meta-webindexer?

Allow it if you want citations in Meta AI.

Sources

Related AI crawlers

OAI-SearchBot · Claude-SearchBot · PerplexityBot · Googlebot · Bingbot · Applebot · meta-externalagent · meta-externalfetcher

Facts checked against Meta docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me