OpenAI runs three bots that site owners usually ask about, and each has a different job. GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot decides whether your pages can appear in ChatGPT search. ChatGPT-User visits a page when a person in ChatGPT asks for it. OpenAI says each robots.txt setting is independent, so you can block one and allow another. The most common choice for product sites is to allow OAI-SearchBot and decide separately about GPTBot.
The three OpenAI bots at a glance
| Bot | What it does | Follows robots.txt? | If you block it | Published IP list |
|---|---|---|---|---|
| GPTBot | Crawls content that may be used to train OpenAI's generative AI foundation models. | Yes | OpenAI treats it as a signal that your content should not be used to train its foundation models. ChatGPT search is not affected. | openai.com/gptbot.json |
| OAI-SearchBot | Surfaces websites in ChatGPT search results. | Yes | Your pages are not shown in ChatGPT search answers, though they can still appear as navigational links. | openai.com/searchbot.json |
| ChatGPT-User | Visits a page when a ChatGPT user or a custom GPT asks a question that needs it, and handles GPT Actions. It does not crawl automatically. | OpenAI says robots.txt rules may not apply, because the visits are user-initiated. | The rule may not apply. It has no effect on ChatGPT search, which OpenAI says ChatGPT-User is not used for. | openai.com/chatgpt-user.json |
GPTBot: model training
GPTBot is OpenAI's training crawler. OpenAI says disallowing it signals that your site's content should not be used in training its generative AI foundation models. It is not the bot behind ChatGPT search, so blocking it does not take you out of ChatGPT's search answers. Whether to block it is a business decision about training data, covered in detail in should you block GPTBot?
OAI-SearchBot: ChatGPT search
OAI-SearchBot is the bot that matters for being cited in ChatGPT search. OpenAI says sites that opt out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links, and recommends allowing it in robots.txt and allowing requests from its published IP ranges. After you change robots.txt, OpenAI says it takes about 24 hours for its search systems to adjust.
If you allow both GPTBot and OAI-SearchBot, OpenAI says it may use the results of just one crawl for both purposes to avoid crawling twice. If you block GPTBot, that rule still stands: the settings are independent.
ChatGPT-User: user-triggered visits
ChatGPT-User is not a crawler in the usual sense. OpenAI says it is used for certain user actions in ChatGPT and custom GPTs, such as visiting a page when a user asks a question, and for interacting with external applications through GPT Actions. It is not used to crawl the web automatically and not used to decide whether content appears in search. Because the visits are user-initiated, OpenAI says robots.txt rules may not apply to it.
What blocking each one changes
| If you disallow | Training | ChatGPT search | User-triggered visits |
|---|---|---|---|
| GPTBot only | Opted out | Unchanged | Unchanged |
| OAI-SearchBot only | Unchanged | Not shown in search answers (navigational links still possible) | Unchanged |
| ChatGPT-User only | Unchanged | Unchanged (not used for search) | OpenAI says the rule may not apply |
| All three | Opted out | Not shown in search answers | OpenAI says the rule may not apply |
Should I block GPTBot?
Block GPTBot if you don't want your content used to train OpenAI's models, for example when the content itself is what you sell. Leave it allowed if you want future models to know your product. Either way, keep OAI-SearchBot allowed if you want to be cited in ChatGPT search. The trade-offs, and what other sites do, are in should you block GPTBot?
robots.txt recipes
Allow ChatGPT search, block training (the usual choice for sites that opt out of training):
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: /
Block training for one section only, for example paid content, and keep the rest available:
User-agent: GPTBot Disallow: /members/ Disallow: /courses/ User-agent: OAI-SearchBot Allow: /
Opt out of both training and ChatGPT search:
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: /
Because OpenAI says robots.txt may not apply to ChatGPT-User, a Disallow for it is not a reliable block. Stopping those visits would take a server or CDN rule, and that also stops ChatGPT from opening pages your users ask it about. For the full file, including other vendors and CMS setup, see robots.txt for AI crawlers, or build one with the AI robots.txt generator.
How to verify OpenAI's bots
Anyone can send a GPTBot user agent, so check the source IP against the list OpenAI publishes for that bot: gptbot.json, searchbot.json and chatgpt-user.json. OpenAI doesn't document a reverse DNS check. The user agents OpenAI publishes (version numbers may change):
GPTBot: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot OAI-SearchBot: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot ChatGPT-User: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
OpenAI also says that when GPTBot and OAI-SearchBot fetch robots.txt, the user agent may include an extra robots.txt marker, which helps you tell those requests apart in your logs. To list the current ranges for one bot:
curl -s https://openai.com/searchbot.json | jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix'
Common mistakes
- Blocking GPTBot to get out of ChatGPT search. GPTBot is for training; ChatGPT search is controlled by OAI-SearchBot.
- Blocking OAI-SearchBot together with GPTBot when you only meant to opt out of training. That also removes you from ChatGPT search answers.
- Disallowing ChatGPT-User to leave ChatGPT search. OpenAI says it isn't used to decide what appears in search.
- Allowing OAI-SearchBot in robots.txt while a firewall or CDN bot rule blocks OpenAI's IP ranges. The robots.txt rule doesn't help if the request never reaches your server.
- Expecting changes to show up immediately. OpenAI says search takes about 24 hours to adjust to a robots.txt change.
Check what OpenAI's bots actually get
The AI crawler checker requests your site as GPTBot, OAI-SearchBot, ChatGPT-User and other AI bots, so it catches CDN and firewall blocks that robots.txt doesn't show. The free scan below checks your robots.txt rules for AI search bots, plus the readable content, structured data and llms.txt that help assistants describe you.