AI crawler list · Perplexity · User-triggered fetcher

Perplexity-User

Visits a page when a Perplexity user's question needs it and links to that page in the answer. Not used for general crawling or training.

Facts checked against Perplexity docs on Oct 20, 2026.

Operator

Perplexity

Used for

Perplexity (user requests)

Type

User-triggered fetcher

Obeys robots.txt

Not reliably

Is Perplexity-User allowed on your site?

Reads your robots.txt and requests your page as each AI bot. Free, no signup.

Should you allow it?

Leave it allowed. It only fetches pages users are asking about, and links back to you.

If you block it: Little effect via robots.txt: Perplexity says this fetcher generally ignores it.

Perplexity says this fetcher generally ignores robots.txt because a user requested the fetch.

robots.txt rules

Allow Perplexity-User

User-agent: Perplexity-User
Allow: /

Block Perplexity-User

User-agent: Perplexity-User
Disallow: /

Disallow Perplexity-User from one folder only

User-agent: Perplexity-User
Disallow: /private/
Allow: /

Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.

This fetcher does not reliably follow robots.txt, so a Disallow rule may have little effect.

Same company, different bots: Perplexity

Perplexity uses separate user agents for separate jobs. Blocking one does not block the others.

Bot	Purpose	Obeys robots.txt	If you block it
`PerplexityBot` AI search index	Perplexity search	Yes	Your site is no longer surfaced and linked in Perplexity search results.
`Perplexity-User` User-triggered fetcher	Perplexity (user requests)	Not reliably	Little effect via robots.txt: Perplexity says this fetcher generally ignores it.

User agent string

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

Versions in the string can change. Match on the token Perplexity-User, and verify real traffic as shown below rather than trusting the header.

How to verify it's really Perplexity-User

Find its requests in your server logs (the path to your access log depends on your host):

grep -i "Perplexity-User" access.log

Perplexity recommends combining user agent matching with its published IP ranges, and refreshing the ranges regularly because they change.

Published IP list: www.perplexity.com/perplexity-user.json

Check one IP against the list (prints True or False):

curl -s https://www.perplexity.com/perplexity-user.json | python3 -c "import sys,re,ipaddress as a; ip=a.ip_address(sys.argv[1]); print(any(ip in a.ip_network(c,strict=False) for c in re.findall(r'[0-9a-fA-F:.]+/[0-9]+', sys.stdin.read())))" 203.0.113.7

Common mistakes

Expecting a robots.txt Disallow to stop it. Perplexity says it generally ignores robots.txt; to stay out of Perplexity search results, PerplexityBot is the token that matters.
Writing a User-agent: Perplexity-User group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the Perplexity-User group.
Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
Trusting the user agent header alone. Anyone can send "Perplexity-User" in a request; verify the source IP as shown above before allowlisting or blocking.

FAQ

What is Perplexity-User?

Visits a page when a Perplexity user's question needs it and links to that page in the answer. Not used for general crawling or training.

Does Perplexity-User respect robots.txt?

Not reliably. Perplexity says user-requested fetches may not follow robots.txt.

What happens if I block Perplexity-User?

Little effect via robots.txt: Perplexity says this fetcher generally ignores it.

Should I block Perplexity-User?

Leave it allowed. It only fetches pages users are asking about, and links back to you.

Is Perplexity-User used for training?

No. Perplexity says it is not used for web crawling or to collect content for training AI foundation models.

What is the difference between PerplexityBot and Perplexity-User?

PerplexityBot crawls to surface and link websites in Perplexity search results and follows robots.txt. Perplexity-User fetches a page live when a user's question needs it and generally ignores robots.txt.

Sources

Related AI crawlers

ChatGPT-User · OAI-AdsBot · Claude-User · PerplexityBot · Google-CloudVertexBot · meta-externalfetcher · Amzn-User · MistralAI-User

Facts checked against Perplexity docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me