AI crawler list · Mistral AI · AI search index
MistralAI-Index
Indexes web content for Mistral search, which helps answer questions in its Vibe assistant. Mistral says the content is not used for generative AI training of any kind.
Facts checked against Mistral AI docs on Oct 20, 2026.
Is MistralAI-Index allowed on your site?
Reads your robots.txt and requests your page as each AI bot. Free, no signup.
Should you allow it?
Allow it.
If you block it: Your content is not indexed for Mistral search.
robots.txt rules
User-agent: MistralAI-Index Allow: /
User-agent: MistralAI-Index Disallow: /
User-agent: MistralAI-Index Disallow: /private/ Allow: /
Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.
Same company, different bots: Mistral AI
Mistral AI uses separate user agents for separate jobs. Blocking one does not block the others.
| Bot | Purpose | Obeys robots.txt | If you block it |
|---|---|---|---|
MistralAI-UserUser-triggered fetcher | Mistral Vibe (user requests) | Yes | Mistral's assistant cannot visit your pages to answer users' questions. |
MistralAI-IndexAI search index | Mistral search | Yes | Your content is not indexed for Mistral search. |
MistralAI-TrainingAI training crawler | Mistral model training | Yes | Your content is not collected for Mistral's training datasets. |
User-agent: MistralAI-User User-agent: MistralAI-Index Allow: / User-agent: MistralAI-Training Disallow: /
User agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)
Versions in the string can change. Match on the token MistralAI-Index, and verify real traffic as shown below rather than trusting the header.
How to verify it's really MistralAI-Index
Find its requests in your server logs (the path to your access log depends on your host):
grep -i "MistralAI-Index" access.log
Mistral publishes IP lists for MistralAI-User and MistralAI-Index.
Published IP list: mistral.ai/mistralai-index-ips.json
Check one IP against the list (prints True or False):
curl -s https://mistral.ai/mistralai-index-ips.json | python3 -c "import sys,re,ipaddress as a; ip=a.ip_address(sys.argv[1]); print(any(ip in a.ip_network(c,strict=False) for c in re.findall(r'[0-9a-fA-F:.]+/[0-9]+', sys.stdin.read())))" 203.0.113.7
Common mistakes
- Writing a User-agent: MistralAI-Index group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the MistralAI-Index group.
- Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
- Trusting the user agent header alone. Anyone can send "MistralAI-Index" in a request; verify the source IP as shown above before allowlisting or blocking.
FAQ
Indexes web content for Mistral search, which helps answer questions in its Vibe assistant. Mistral says the content is not used for generative AI training of any kind.
Yes, according to Mistral AI.
Your content is not indexed for Mistral search.
Allow it.
Sources
- Official documentation from Mistral AI
- Published IP addresses for MistralAI-Index
- Cloudflare bots changelog (AI bot policy defaults)
Related AI crawlers
OAI-SearchBot · Claude-SearchBot · PerplexityBot · Googlebot · Bingbot · Applebot · meta-webindexer · Amzn-SearchBot
Facts checked against Mistral AI docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me