AI crawler list · OpenAI · AI training crawler

GPTBot

Crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing it tells OpenAI not to use your content for training.

Facts checked against OpenAI docs on Oct 20, 2026.

Operator

OpenAI

Used for

OpenAI model training

Type

AI training crawler

Obeys robots.txt

Yes

Is GPTBot allowed on your site?

Reads your robots.txt and requests your page as each AI bot. Free, no signup.

Should you allow it?

Your choice. Allow it if you want future models to know your brand; block it if you do not want your content used for training.

If you block it: Signals OpenAI not to use your content to train its foundation models. No effect on ChatGPT search.

Blocking GPTBot does not remove you from ChatGPT search. That is OAI-SearchBot.

When fetching robots.txt, OpenAI says it may add a "robots.txt" marker to GPTBot's user agent, so match on the GPTBot token, not the full string.

robots.txt rules

Allow GPTBot

User-agent: GPTBot
Allow: /

Block GPTBot

User-agent: GPTBot
Disallow: /

Disallow GPTBot from one folder only

User-agent: GPTBot
Disallow: /private/
Allow: /

Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.

Same company, different bots: OpenAI

OpenAI uses separate user agents for separate jobs. Blocking one does not block the others.

Bot	Purpose	Obeys robots.txt	If you block it
`OAI-SearchBot` AI search index	ChatGPT search	Yes	Your pages stop appearing in ChatGPT search answers (they can still show up as navigational links).
`ChatGPT-User` User-triggered fetcher	ChatGPT (user actions and GPT Actions)	Not reliably	A robots.txt rule may not apply, OpenAI says. It has no effect on ChatGPT search eligibility.
`GPTBot` AI training crawler	OpenAI model training	Yes	Signals OpenAI not to use your content to train its foundation models. No effect on ChatGPT search.
`OAI-AdsBot` User-triggered fetcher	ChatGPT ads	Not documented	OpenAI's docs don't describe a robots.txt opt-out. It only visits pages submitted as ChatGPT ads, to review them.

Allow OpenAI search and user fetches, opt out of OpenAI training

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
Allow: /

User-agent: GPTBot
Disallow: /

User agent string

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Versions in the string can change. Match on the token GPTBot, and verify real traffic as shown below rather than trusting the header.

How to verify it's really GPTBot

Find its requests in your server logs (the path to your access log depends on your host):

grep -i "GPTBot" access.log

OpenAI publishes a separate IP list for each of its bots and recommends allowing requests from those ranges. It does not document a reverse DNS check.

Published IP list: openai.com/gptbot.json

Check one IP against the list (prints True or False):

curl -s https://openai.com/gptbot.json | python3 -c "import sys,re,ipaddress as a; ip=a.ip_address(sys.argv[1]); print(any(ip in a.ip_network(c,strict=False) for c in re.findall(r'[0-9a-fA-F:.]+/[0-9]+', sys.stdin.read())))" 203.0.113.7

Common mistakes

Blocking OAI-SearchBot along with GPTBot when you only meant to opt out of training. That also removes you from ChatGPT search answers.
Writing a User-agent: GPTBot group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the GPTBot group.
Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
Trusting the user agent header alone. Anyone can send "GPTBot" in a request; verify the source IP as shown above before allowlisting or blocking.

FAQ

What is GPTBot?

Crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing it tells OpenAI not to use your content for training.

Does GPTBot respect robots.txt?

Yes, according to OpenAI.

What happens if I block GPTBot?

Signals OpenAI not to use your content to train its foundation models. No effect on ChatGPT search.

Should I block GPTBot?

Your choice. Allow it if you want future models to know your brand; block it if you do not want your content used for training.

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot only collects content that may be used for training. ChatGPT search uses OAI-SearchBot, and pages users ask about are fetched by ChatGPT-User. You can block GPTBot and still be cited in ChatGPT search.

How do I block GPTBot but stay in ChatGPT search?

Add a User-agent: GPTBot group with Disallow: / and a User-agent: OAI-SearchBot group with Allow: /. OpenAI controls the two separately.

Sources

Related AI crawlers

OAI-SearchBot · ChatGPT-User · OAI-AdsBot · ClaudeBot · meta-externalagent · Amazonbot · MistralAI-Training · CCBot

Facts checked against OpenAI docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me