AI crawler list · Perplexity · AI search index

PerplexityBot

Surfaces and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models.

Facts checked against Perplexity docs on Oct 20, 2026.

Operator

Perplexity

Used for

Perplexity search

Type

AI search index

Obeys robots.txt

Yes

Is PerplexityBot allowed on your site?

Reads your robots.txt and requests your page as each AI bot. Free, no signup.

Should you allow it?

Allow it. Perplexity cites sources prominently, so this is direct referral traffic.

If you block it: Your site is no longer surfaced and linked in Perplexity search results.

Perplexity says robots.txt changes can take up to 24 hours to take effect.

robots.txt rules

Allow PerplexityBot

User-agent: PerplexityBot
Allow: /

Block PerplexityBot

User-agent: PerplexityBot
Disallow: /

Disallow PerplexityBot from one folder only

User-agent: PerplexityBot
Disallow: /private/
Allow: /

Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.

Same company, different bots: Perplexity

Perplexity uses separate user agents for separate jobs. Blocking one does not block the others.

Bot	Purpose	Obeys robots.txt	If you block it
`PerplexityBot` AI search index	Perplexity search	Yes	Your site is no longer surfaced and linked in Perplexity search results.
`Perplexity-User` User-triggered fetcher	Perplexity (user requests)	Not reliably	Little effect via robots.txt: Perplexity says this fetcher generally ignores it.

User agent string

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Versions in the string can change. Match on the token PerplexityBot, and verify real traffic as shown below rather than trusting the header.

How to verify it's really PerplexityBot

Find its requests in your server logs (the path to your access log depends on your host):

grep -i "PerplexityBot" access.log

Perplexity recommends combining user agent matching with its published IP ranges, and refreshing the ranges regularly because they change.

Published IP list: www.perplexity.com/perplexitybot.json

Check one IP against the list (prints True or False):

curl -s https://www.perplexity.com/perplexitybot.json | python3 -c "import sys,re,ipaddress as a; ip=a.ip_address(sys.argv[1]); print(any(ip in a.ip_network(c,strict=False) for c in re.findall(r'[0-9a-fA-F:.]+/[0-9]+', sys.stdin.read())))" 203.0.113.7

Common mistakes

Allowing it in robots.txt while a WAF or bot-protection rule blocks it. Perplexity says to allow its user agent from its published IP ranges in your WAF too.
Writing a User-agent: PerplexityBot group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the PerplexityBot group.
Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
Trusting the user agent header alone. Anyone can send "PerplexityBot" in a request; verify the source IP as shown above before allowlisting or blocking.

FAQ

What is PerplexityBot?

Surfaces and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models.

Does PerplexityBot respect robots.txt?

Yes, according to Perplexity.

What happens if I block PerplexityBot?

Your site is no longer surfaced and linked in Perplexity search results.

Should I block PerplexityBot?

Allow it. Perplexity cites sources prominently, so this is direct referral traffic.

Is PerplexityBot used to train AI models?

No. Perplexity says PerplexityBot is designed to surface and link websites in search results and is not used to crawl content for AI foundation models.

How long until a PerplexityBot robots.txt change works?

Perplexity says changes can take up to 24 hours to take effect.

How do I allow PerplexityBot through Cloudflare or AWS WAF?

Perplexity recommends a rule that matches the PerplexityBot user agent AND a source IP in its published ranges, with the action set to Allow and a higher priority than your blocking rules. Refresh the IP ranges regularly because they change.

Sources

Related AI crawlers

OAI-SearchBot · Claude-SearchBot · Perplexity-User · Googlebot · Bingbot · Applebot · meta-webindexer · Amzn-SearchBot

Facts checked against Perplexity docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me