AI crawler list · Apple · AI search index

Applebot

Powers search in Spotlight, Siri and Safari. Apple says crawled data may also help train its foundation models and supply context for AI answers.

Facts checked against Apple docs on Oct 20, 2026.

Operator

Apple

Used for

Siri, Spotlight and Safari search

Type

AI search index

Obeys robots.txt

Yes

Is Applebot allowed on your site?

Reads your robots.txt and requests your page as each AI bot. Free, no signup.

Should you allow it?

Allow it. To opt out of Apple model training only, disallow Applebot-Extended instead.

If you block it: Your content can't be crawled for Spotlight, Siri and Safari search results.

The user agent is Apple's desktop example. Apple says the browser version it advertises changes over time, so match on the Applebot token.

If your robots.txt has no Applebot rules but has Googlebot rules, Applebot follows the Googlebot rules.

Applebot ignores Crawl-delay.

To keep a page out of AI answer context while staying in search, Apple says to use the nosnippet meta tag. It also accepts the X-Robots-Tag header, for example X-Robots-Tag: applebot: nosnippet.

Apple says pages marked with schema.org isAccessibleForFree: false can still appear in search, but Applebot won't use that content as context for AI answers.

robots.txt rules

Allow Applebot

User-agent: Applebot
Allow: /

Block Applebot

User-agent: Applebot
Disallow: /

Disallow Applebot from one folder only

User-agent: Applebot
Disallow: /private/
Allow: /

Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.

Same company, different bots: Apple

Apple uses separate user agents for separate jobs. Blocking one does not block the others.

Bot	Purpose	Obeys robots.txt	If you block it
`Applebot` AI search index	Siri, Spotlight and Safari search	Yes	Your content can't be crawled for Spotlight, Siri and Safari search results.
`Applebot-Extended` Control token (does not crawl)	Apple model training	Token only	Apple won't use Applebot-crawled content to train its generative models. Siri, Spotlight and Safari search are unaffected.

Allow Apple search and user fetches, opt out of Apple training

User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Disallow: /

User agent string

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

Versions in the string can change. Match on the token Applebot, and verify real traffic as shown below rather than trusting the header.

How to verify it's really Applebot

Find its requests in your server logs (the path to your access log depends on your host):

grep -i "Applebot" access.log

Reverse DNS check (Apple documents this): look up the IP from your log, confirm the hostname ends in applebot.apple.com, then look up that hostname and confirm it returns the same IP.

host 203.0.113.7          # replace with the IP from your log
host <hostname-from-above>  # must return the same IP

Published IP list: search.developer.apple.com/applebot.json

Check one IP against the list (prints True or False):

curl -s https://search.developer.apple.com/applebot.json | python3 -c "import sys,re,ipaddress as a; ip=a.ip_address(sys.argv[1]); print(any(ip in a.ip_network(c,strict=False) for c in re.findall(r'[0-9a-fA-F:.]+/[0-9]+', sys.stdin.read())))" 203.0.113.7

Common mistakes

Blocking JavaScript or CSS files. Apple says Applebot may render pages and rendering can fail when those resources are blocked.
Forgetting that Googlebot rules apply. With no Applebot group, a Disallow you wrote for Googlebot also applies to Applebot.
Writing a User-agent: Applebot group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the Applebot group.
Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
Trusting the user agent header alone. Anyone can send "Applebot" in a request; verify the source IP as shown above before allowlisting or blocking.

FAQ

What is Applebot?

Powers search in Spotlight, Siri and Safari. Apple says crawled data may also help train its foundation models and supply context for AI answers.

Does Applebot respect robots.txt?

Yes, according to Apple.

What happens if I block Applebot?

Your content can't be crawled for Spotlight, Siri and Safari search results.

Should I block Applebot?

Allow it. To opt out of Apple model training only, disallow Applebot-Extended instead.

Sources

Related AI crawlers

OAI-SearchBot · Claude-SearchBot · PerplexityBot · Googlebot · Bingbot · Applebot-Extended · meta-webindexer · Amzn-SearchBot

Facts checked against Apple docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me