AI crawler list · OpenAI · AI search index

OAI-SearchBot

Crawls the web so pages can be surfaced and linked in ChatGPT search answers. OpenAI says sites that opt out will not be shown in ChatGPT search answers, though they can still appear as navigational links.

Facts checked against OpenAI docs on Oct 20, 2026.

Operator

OpenAI

Used for

ChatGPT search

Type

AI search index

Obeys robots.txt

Yes

Is OAI-SearchBot allowed on your site?

Reads your robots.txt and requests your page as each AI bot. Free, no signup.

Should you allow it?

Allow it. This is the bot that decides whether ChatGPT search can cite you.

If you block it: Your pages stop appearing in ChatGPT search answers (they can still show up as navigational links).

OpenAI says robots.txt changes for search can take about 24 hours to take effect.

Separate from GPTBot: you can block training and still allow ChatGPT search.

When fetching robots.txt, OpenAI says it may add a "robots.txt" marker to the user agent (e.g. "compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot"). Match on the OAI-SearchBot token, not the full string.

robots.txt rules

Allow OAI-SearchBot

User-agent: OAI-SearchBot
Allow: /

Block OAI-SearchBot

User-agent: OAI-SearchBot
Disallow: /

Disallow OAI-SearchBot from one folder only

User-agent: OAI-SearchBot
Disallow: /private/
Allow: /

Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.

Same company, different bots: OpenAI

OpenAI uses separate user agents for separate jobs. Blocking one does not block the others.

Bot	Purpose	Obeys robots.txt	If you block it
`OAI-SearchBot` AI search index	ChatGPT search	Yes	Your pages stop appearing in ChatGPT search answers (they can still show up as navigational links).
`ChatGPT-User` User-triggered fetcher	ChatGPT (user actions and GPT Actions)	Not reliably	A robots.txt rule may not apply, OpenAI says. It has no effect on ChatGPT search eligibility.
`GPTBot` AI training crawler	OpenAI model training	Yes	Signals OpenAI not to use your content to train its foundation models. No effect on ChatGPT search.
`OAI-AdsBot` User-triggered fetcher	ChatGPT ads	Not documented	OpenAI's docs don't describe a robots.txt opt-out. It only visits pages submitted as ChatGPT ads, to review them.

Allow OpenAI search and user fetches, opt out of OpenAI training

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
Allow: /

User-agent: GPTBot
Disallow: /

User agent string

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

Versions in the string can change. Match on the token OAI-SearchBot, and verify real traffic as shown below rather than trusting the header.

How to verify it's really OAI-SearchBot

Find its requests in your server logs (the path to your access log depends on your host):

grep -i "OAI-SearchBot" access.log

OpenAI publishes a separate IP list for each of its bots and recommends allowing requests from those ranges. It does not document a reverse DNS check.

Published IP list: openai.com/searchbot.json

Check one IP against the list (prints True or False):

curl -s https://openai.com/searchbot.json | python3 -c "import sys,re,ipaddress as a; ip=a.ip_address(sys.argv[1]); print(any(ip in a.ip_network(c,strict=False) for c in re.findall(r'[0-9a-fA-F:.]+/[0-9]+', sys.stdin.read())))" 203.0.113.7

Common mistakes

Blocking GPTBot to get out of ChatGPT search. GPTBot is training only; ChatGPT search is controlled by OAI-SearchBot.
Allowing it in robots.txt but blocking OpenAI's IP ranges in a firewall. OpenAI recommends allowing both the robots.txt token and requests from its published IP ranges.
Writing a User-agent: OAI-SearchBot group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the OAI-SearchBot group.
Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
Trusting the user agent header alone. Anyone can send "OAI-SearchBot" in a request; verify the source IP as shown above before allowlisting or blocking.

FAQ

What is OAI-SearchBot?

Does OAI-SearchBot respect robots.txt?

Yes, according to OpenAI.

What happens if I block OAI-SearchBot?

Your pages stop appearing in ChatGPT search answers (they can still show up as navigational links).

Should I block OAI-SearchBot?

Allow it. This is the bot that decides whether ChatGPT search can cite you.

What is the difference between GPTBot and OAI-SearchBot?

OpenAI controls them separately in robots.txt. GPTBot crawls content that may be used to train OpenAI's foundation models. OAI-SearchBot surfaces websites in ChatGPT search. You can allow OAI-SearchBot and disallow GPTBot to stay in ChatGPT search while opting out of training.

How long does an OAI-SearchBot robots.txt change take to work?

OpenAI says it takes roughly 24 hours for its systems to adjust after you change robots.txt rules for search.

If I allow both GPTBot and OAI-SearchBot, does OpenAI crawl my site twice?

Not necessarily. OpenAI says that if a site allows both bots, it may use the results from just one crawl for both purposes to avoid duplicate crawling.

Sources

Related AI crawlers

ChatGPT-User · GPTBot · OAI-AdsBot · Claude-SearchBot · PerplexityBot · Googlebot · Bingbot · Applebot

Facts checked against OpenAI docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me