Guides · Updated 2026-10-27

robots.txt for AI crawlers: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot

AI companies run several crawlers with different jobs. Some fetch pages to answer a user right now (and can cite you), others collect training data. Your robots.txt can treat them differently, so you can stay visible in AI answers while opting out of model training. (Per-bot details: AI bots directory.)

Search and user-triggered crawlers (allow these to get cited)

  • OAI-SearchBot: OpenAI says it is for search. Sites that opt out are left out of ChatGPT search answers.
  • ChatGPT-User: fetches a page when a ChatGPT user asks. OpenAI says robots.txt rules may not apply because the visit is user-initiated.
  • PerplexityBot and Perplexity-User: Perplexity's search crawler and its live fetcher. Perplexity says Perplexity-User generally ignores robots.txt.
  • Claude-SearchBot and Claude-User: Claude's search indexing and live fetches. Anthropic says its bots honor robots.txt.
  • Googlebot and Bingbot: classic search, which also feeds Google AI Overviews and Bing Copilot.

Training crawlers (your choice)

  • GPTBot (OpenAI)
  • ClaudeBot (Anthropic)
  • Google-Extended (a robots.txt control token for Gemini training and grounding, not a separate crawler)
  • Applebot-Extended (Apple)
  • CCBot (Common Crawl, used by many models)
  • meta-externalagent (Meta)

Decision table: block training, keep AI search

If you want torobots.txt ruleWhat the vendor says
Appear in ChatGPT searchAllow OAI-SearchBotOpenAI: OAI-SearchBot is for search; opted-out sites are left out of ChatGPT search answers. Changes take about 24 hours.
Keep content out of OpenAI trainingDisallow GPTBotOpenAI: disallowing GPTBot means content should not be used to train its foundation models. It is separate from search.
Appear in Claude answersAllow Claude-SearchBot and Claude-UserAnthropic: blocking either may reduce your visibility in Claude's search results and answers to users.
Keep content out of Anthropic trainingDisallow ClaudeBot (on every subdomain)Anthropic: its bots honor robots.txt, and you must add the rule to each subdomain you want to opt out.
Appear in PerplexityAllow PerplexityBotPerplexity: PerplexityBot surfaces sites in its search results and is not used to train foundation models.
Appear in Google AI Overviews and AI ModeAllow GooglebotGoogle: Googlebot's robots.txt rules are the control for Search, including its AI features. Use nosnippet, max-snippet or noindex to limit what is shown.
Limit Gemini training and groundingDisallow Google-ExtendedGoogle points to Google-Extended to limit AI training and grounding in its other systems; Search is controlled by Googlebot.

Rules for user-triggered fetchers (ChatGPT-User, Perplexity-User) have little effect, because both vendors say these fetches may not follow robots.txt. Claude-User is the exception: Anthropic says its bots honor robots.txt.

A sensible default

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

To also opt out of training, add groups for the training crawlers. AI search citations keep working:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Or pick bots one by one in the free AI robots.txt generator.

Cloudflare and CDN settings that override robots.txt

robots.txt is a request; a CDN or firewall rule is enforced. If your robots.txt allows a bot but a Cloudflare setting blocks it, the bot is blocked. If AI search never cites you, check these first:

  • Cloudflare AI bot policies (Security Settings, Configure AI bot policies): separate Block, Block on pages with ads, or Allow choices for Search, Agent and Training bots (Training also has Disallow, used with Bot Preference Sync). Cloudflare defines Agent as automated activity acting in real time for a person, "such as chat fetch bots and browser-use agents".
  • New Cloudflare defaults: Cloudflare says that from September 15, 2026, new domains block Training and Agent bots on pages that display ads, while Search stays allowed. If you show ads, live fetches by AI assistants may be blocked unless you change this.
  • Mixed-purpose crawlers: Cloudflare's AI bot policies page says crawlers used for both Search and Training are blocked by every configuration that blocks AI training. Its robots.txt page says the Training Disallow option leaves mixed-use crawlers out of its edge block.
  • The older Cloudflare "Block AI bots" setting blocks verified bots that crawl for AI training, plus some unverified bots that behave similarly. Cloudflare marks it as deprecating on September 15, 2026, in favor of the AI bot policies.
  • Cloudflare Bot Preference Sync (the managed robots.txt): turned on in the AI bot policies, it writes your policies into robots.txt, but only when at least one of Search, Agent or Training is set to Block, Block on pages with ads or (for Training) Disallow. With every policy on Allow, your robots.txt is served unchanged. Setting Training to Disallow adds Disallow groups for training crawlers such as GPTBot, ClaudeBot and Google-Extended, plus a Content-Signal line, and also blocks those crawlers at Cloudflare's edge (Cloudflare says mixed-use crawlers are excluded from this edge block). Search and Agent add their own rules when set to Block or Block on pages with ads. Cloudflare prepends these rules before your existing robots.txt. Open yourdomain.com/robots.txt to see what bots actually receive.

Other hosts and WAFs have similar bot-blocking options. The AI crawler checker requests your site as each AI bot, so it catches blocks that robots.txt alone doesn't show.

WordPress

WordPress generates /robots.txt itself (the do_robots function). To add AI crawler rules from a theme or plugin, use the robots_txt filter, which receives the generated text and returns the final output:

add_filter( 'robots_txt', function ( $output, $public ) {
    $output .= "\nUser-agent: GPTBot\nDisallow: /\n";
    $output .= "\nUser-agent: OAI-SearchBot\nAllow: /\n";
    return $output;
}, 10, 2 );

No code? The free RankRoot WordPress plugin adds AI crawler rules to robots.txt and publishes llms.txt.

Webflow

  • Go to Site settings, then SEO, then Indexing.
  • Add your rules (a User-agent line, then Allow or Disallow).
  • Click Save and publish the site.
  • The same Indexing panel has a Content-Signal setting. Webflow notes it is based on a proposed extension to the robots.txt standard, not yet an accepted standard.

Shopify

Create a file named robots.txt.liquid in your theme's Templates folder. Shopify says this template replaces its generated robots.txt, so save a copy of your current /robots.txt first. Keep Shopify's default Liquid loop and add your AI crawler groups outside it:

{% comment %} Keep Shopify's default robots loop above this line {% endcomment %}

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Ghost, GitHub Pages, Carrd, Hostinger, Notion, Netlify and Vercel

Ghost: add a robots.txt file to the root of your theme and upload the new theme version. Ghost says it overrides the default file, so keep the default Sitemap and Disallow lines. Steps in the Ghost guide.

GitHub Pages: commit a plain robots.txt to the root of what you publish. A project site at owner.github.io/repository can't have its own, because crawlers only read /robots.txt at the top of the host. See the GitHub Pages guide.

Carrd: robots.txt is editable on Pro Plus or higher under Dashboard > Sites > Manage > Files. Clearing a file's contents removes it. Steps in the Carrd guide.

Hostinger AI Builder generates robots.txt automatically for sites on a custom domain, and its help center doesn't describe a way to edit it. Use a page's Hide page from search results setting (a noindex tag) to keep one page out. See the Hostinger AI Builder guide, or all website builders.

Notion Sites: Notion's help center documents no way to edit robots.txt. The only documented control is Share > Publish > Search engine indexing > Discoverable on the web. See the Notion Sites guide.

Netlify and Vercel serve the robots.txt your build outputs: put it in the publish directory or your framework's static folder (public/ in Next.js), or generate it with app/robots.ts. On Vercel, also check Firewall > Rules > Bot Management: the AI Bots ruleset is off by default, but set to Deny it blocks AI search and user-requested fetches along with training bots, whatever robots.txt says. See the Netlify guide and the Vercel guide.

Cloudflare Pages and Workers serve the robots.txt in your static assets folder. If Bot Preference Sync is on, Cloudflare prepends its managed rules to it, and the AI bot policies above still block at the edge whatever the file says. See the Cloudflare Pages & Workers guide.

Next.js (App Router)

Add app/robots.ts. Next.js turns the returned object into /robots.txt, with one group per user agent:

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: ['OAI-SearchBot', 'ChatGPT-User', 'PerplexityBot', 'Claude-SearchBot', 'Claude-User'], allow: '/' },
      { userAgent: ['GPTBot', 'ClaudeBot', 'Google-Extended', 'CCBot'], disallow: '/' },
      { userAgent: '*', allow: '/' },
    ],
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

A static app/robots.txt file works too. Either way, a CDN rule can still block bots that robots.txt allows, so run the AI crawler checker after you deploy.

Sources

Check your site now

Just your website. Free, no signup.

FAQ

Will blocking AI crawlers hurt my Google rankings?

Blocking Google-Extended doesn't affect Google Search. Blocking Googlebot does, and it also removes you from AI Overviews and AI Mode. Blocking OpenAI, Anthropic or Perplexity crawlers only affects those products.

Do AI crawlers respect robots.txt?

OpenAI, Anthropic and Perplexity document that their crawlers follow robots.txt. User-triggered fetchers differ: OpenAI says robots.txt may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores it, while Anthropic says its bots, including Claude-User, honor it.

Can I block AI training but stay in ChatGPT search?

Yes. Disallow GPTBot and allow OAI-SearchBot. OpenAI controls them separately, and says changes take about 24 hours to apply to search.

My robots.txt allows AI bots. Why am I still blocked?

Usually a CDN or firewall rule. On Cloudflare, check the AI bot policies under Security Settings: since September 15, 2026, new domains block Training and Agent bots on pages with ads by default. On Vercel, check whether the AI Bots managed ruleset is set to Deny. The AI crawler checker shows which bots actually get through.

Does Cloudflare's managed robots.txt replace my file?

No. Cloudflare says that if your site already serves a robots.txt (an HTTP 200 response), Bot Preference Sync prepends its managed rules before your content. If you have none, it creates one. It only adds rules when at least one AI bot policy blocks or disallows; with every policy on Allow, your file is served unchanged. Its managed rules currently add use=reference to the Content-Signal line, an extension Cloudflare says it is testing.

More guides