# robots.txt for AI crawlers: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot

> Which AI crawlers to allow or block, a decision table for search vs training, Cloudflare settings that override robots.txt, and setup for WordPress, Webflow, Shopify and Next.js.

Canonical page: https://getrankroot.com/guides/robots-txt-ai-crawlers · Updated 2026-10-27

AI companies run several crawlers with different jobs. Some fetch pages to answer a user right now (and can cite you), others collect training data. Your robots.txt can treat them differently, so you can stay visible in AI answers while opting out of model training. (Per-bot details: [AI bots directory](https://getrankroot.com/ai-bots).)

## Search and user-triggered crawlers (allow these to get cited)

- [OAI-SearchBot](https://getrankroot.com/ai-bots/oai-searchbot): OpenAI says it is for search. Sites that opt out are left out of ChatGPT search answers.
- [ChatGPT-User](https://getrankroot.com/ai-bots/chatgpt-user): fetches a page when a ChatGPT user asks. OpenAI says robots.txt rules may not apply because the visit is user-initiated.
- PerplexityBot and Perplexity-User: Perplexity's search crawler and its live fetcher. Perplexity says Perplexity-User generally ignores robots.txt.
- Claude-SearchBot and Claude-User: Claude's search indexing and live fetches. Anthropic says its bots honor robots.txt.
- Googlebot and Bingbot: classic search, which also feeds Google AI Overviews and Bing Copilot.

## Training crawlers (your choice)

- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended (a robots.txt control token for Gemini training and grounding, not a separate crawler)
- Applebot-Extended (Apple)
- CCBot (Common Crawl, used by many models)
- meta-externalagent (Meta)

## Decision table: block training, keep AI search

| If you want to | robots.txt rule | What the vendor says |
| --- | --- | --- |
| Appear in ChatGPT search | Allow OAI-SearchBot | OpenAI: OAI-SearchBot is for search; opted-out sites are left out of ChatGPT search answers. Changes take about 24 hours. |
| Keep content out of OpenAI training | Disallow GPTBot | OpenAI: disallowing GPTBot means content should not be used to train its foundation models. It is separate from search. |
| Appear in Claude answers | Allow Claude-SearchBot and Claude-User | Anthropic: blocking either may reduce your visibility in Claude's search results and answers to users. |
| Keep content out of Anthropic training | Disallow ClaudeBot (on every subdomain) | Anthropic: its bots honor robots.txt, and you must add the rule to each subdomain you want to opt out. |
| Appear in Perplexity | Allow PerplexityBot | Perplexity: PerplexityBot surfaces sites in its search results and is not used to train foundation models. |
| Appear in Google AI Overviews and AI Mode | Allow Googlebot | Google: Googlebot's robots.txt rules are the control for Search, including its AI features. Use nosnippet, max-snippet or noindex to limit what is shown. |
| Limit Gemini training and grounding | Disallow Google-Extended | Google points to Google-Extended to limit AI training and grounding in its other systems; Search is controlled by Googlebot. |

Rules for user-triggered fetchers (ChatGPT-User, Perplexity-User) have little effect, because both vendors say these fetches may not follow robots.txt. Claude-User is the exception: Anthropic says its bots honor robots.txt.

## A sensible default

```
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml
```

To also opt out of training, add groups for the training crawlers. AI search citations keep working:

```
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /
```

Or pick bots one by one in the free [AI robots.txt generator](https://getrankroot.com/tools/robots-txt-generator).

## Cloudflare and CDN settings that override robots.txt

robots.txt is a request; a CDN or firewall rule is enforced. If your robots.txt allows a bot but a Cloudflare setting blocks it, the bot is blocked. If AI search never cites you, check these first:

- Cloudflare AI bot policies (Security Settings, Configure AI bot policies): separate Block, Block on pages with ads, or Allow choices for Search, Agent and Training bots (Training also has Disallow, used with Bot Preference Sync). Cloudflare defines Agent as automated activity acting in real time for a person, "such as chat fetch bots and browser-use agents".
- New Cloudflare defaults: Cloudflare says that from September 15, 2026, new domains block Training and Agent bots on pages that display ads, while Search stays allowed. If you show ads, live fetches by AI assistants may be blocked unless you change this.
- Mixed-purpose crawlers: Cloudflare's AI bot policies page says crawlers used for both Search and Training are blocked by every configuration that blocks AI training. Its robots.txt page says the Training Disallow option leaves mixed-use crawlers out of its edge block.
- The older Cloudflare "Block AI bots" setting blocks verified bots that crawl for AI training, plus some unverified bots that behave similarly. Cloudflare marks it as deprecating on September 15, 2026, in favor of the AI bot policies.
- Cloudflare Bot Preference Sync (the managed robots.txt): turned on in the AI bot policies, it writes your policies into robots.txt, but only when at least one of Search, Agent or Training is set to Block, Block on pages with ads or (for Training) Disallow. With every policy on Allow, your robots.txt is served unchanged. Setting Training to Disallow adds Disallow groups for training crawlers such as GPTBot, ClaudeBot and Google-Extended, plus a Content-Signal line, and also blocks those crawlers at Cloudflare's edge (Cloudflare says mixed-use crawlers are excluded from this edge block). Search and Agent add their own rules when set to Block or Block on pages with ads. Cloudflare prepends these rules before your existing robots.txt. Open yourdomain.com/robots.txt to see what bots actually receive.

Other hosts and WAFs have similar bot-blocking options. The [AI crawler checker](https://getrankroot.com/tools/ai-crawler-checker) requests your site as each AI bot, so it catches blocks that robots.txt alone doesn't show.

## WordPress

WordPress generates /robots.txt itself (the do_robots function). To add AI crawler rules from a theme or plugin, use the robots_txt filter, which receives the generated text and returns the final output:

```
add_filter( 'robots_txt', function ( $output, $public ) {
    $output .= "\nUser-agent: GPTBot\nDisallow: /\n";
    $output .= "\nUser-agent: OAI-SearchBot\nAllow: /\n";
    return $output;
}, 10, 2 );
```

No code? The free [RankRoot WordPress plugin](https://getrankroot.com/wordpress) adds AI crawler rules to robots.txt and publishes llms.txt.

## Webflow

- Go to Site settings, then SEO, then Indexing.
- Add your rules (a User-agent line, then Allow or Disallow).
- Click Save and publish the site.
- The same Indexing panel has a Content-Signal setting. Webflow notes it is based on a proposed extension to the robots.txt standard, not yet an accepted standard.

## Shopify

Create a file named robots.txt.liquid in your theme's Templates folder. Shopify says this template replaces its generated robots.txt, so save a copy of your current /robots.txt first. Keep Shopify's default Liquid loop and add your AI crawler groups outside it:

```
{% comment %} Keep Shopify's default robots loop above this line {% endcomment %}

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
```

## Ghost, GitHub Pages, Carrd, Hostinger, Notion, Netlify and Vercel

Ghost: add a robots.txt file to the root of your theme and upload the new theme version. Ghost says it overrides the default file, so keep the default Sitemap and Disallow lines. Steps in the [Ghost guide](https://getrankroot.com/builders/ghost).

GitHub Pages: commit a plain robots.txt to the root of what you publish. A project site at owner.github.io/repository can't have its own, because crawlers only read /robots.txt at the top of the host. See the [GitHub Pages guide](https://getrankroot.com/builders/github-pages).

Carrd: robots.txt is editable on Pro Plus or higher under Dashboard > Sites > Manage > Files. Clearing a file's contents removes it. Steps in the [Carrd guide](https://getrankroot.com/builders/carrd).

Hostinger AI Builder generates robots.txt automatically for sites on a custom domain, and its help center doesn't describe a way to edit it. Use a page's Hide page from search results setting (a noindex tag) to keep one page out. See the [Hostinger AI Builder guide](https://getrankroot.com/builders/hostinger), or [all website builders](https://getrankroot.com/builders).

Notion Sites: Notion's help center documents no way to edit robots.txt. The only documented control is Share > Publish > Search engine indexing > Discoverable on the web. See the [Notion Sites guide](https://getrankroot.com/builders/notion).

Netlify and Vercel serve the robots.txt your build outputs: put it in the publish directory or your framework's static folder (public/ in Next.js), or generate it with app/robots.ts. On Vercel, also check Firewall > Rules > Bot Management: the AI Bots ruleset is off by default, but set to Deny it blocks AI search and user-requested fetches along with training bots, whatever robots.txt says. See the [Netlify guide](https://getrankroot.com/builders/netlify) and the [Vercel guide](https://getrankroot.com/builders/vercel).

Cloudflare Pages and Workers serve the robots.txt in your static assets folder. If Bot Preference Sync is on, Cloudflare prepends its managed rules to it, and the AI bot policies above still block at the edge whatever the file says. See the [Cloudflare Pages & Workers guide](https://getrankroot.com/builders/cloudflare).

## Next.js (App Router)

Add app/robots.ts. Next.js turns the returned object into /robots.txt, with one group per user agent:

```
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: ['OAI-SearchBot', 'ChatGPT-User', 'PerplexityBot', 'Claude-SearchBot', 'Claude-User'], allow: '/' },
      { userAgent: ['GPTBot', 'ClaudeBot', 'Google-Extended', 'CCBot'], disallow: '/' },
      { userAgent: '*', allow: '/' },
    ],
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}
```

A static app/robots.txt file works too. Either way, a CDN rule can still block bots that robots.txt allows, so run the [AI crawler checker](https://getrankroot.com/tools/ai-crawler-checker) after you deploy.

## Sources

- [OpenAI: Overview of OpenAI crawlers (read 2026-10-27)](https://developers.openai.com/api/docs/bots)
- [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (read 2026-10-27)](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity: Perplexity crawlers (read 2026-10-27)](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- [Google: AI features and your website (read 2026-10-27)](https://developers.google.com/search/docs/appearance/ai-features)
- [Cloudflare: Block AI bots and AI bot policies (read 2026-10-27)](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/)
- [Ghost: Modifying robots.txt (read 2026-10-19)](https://ghost.org/help/modifying-robots-txt/)
- [Carrd: Customizing Site Files (read 2026-10-20)](https://carrd.com/docs/sites/customizing-site-files)
- [Notion Help: Publish a website with Notion Sites (read 2026-10-23)](https://www.notion.com/help/public-pages-and-web-publishing)
- [Hostinger: AI Builder SEO settings in agentic mode (read 2026-10-20)](https://www.hostinger.com/support/10771376-hostinger-ai-builder-agentic-mode-seo-settings/)
- [Vercel Docs: Bot Management, AI bots managed ruleset (read 2026-10-21)](https://vercel.com/docs/bot-management)
- [Netlify Docs: Build configuration overview, publish directory (read 2026-10-21)](https://docs.netlify.com/build/configure-builds/overview/)
- [RFC 9309: Robots Exclusion Protocol (read 2026-10-19)](https://www.rfc-editor.org/rfc/rfc9309.html)
- [Cloudflare: robots.txt setting and Bot Preference Sync (read 2026-10-27)](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/)
- [WordPress: do_robots() and the robots_txt filter](https://developer.wordpress.org/reference/functions/do_robots/)
- [Webflow: Set robots.txt rules](https://help.webflow.com/hc/en-us/articles/41954080897683-Set-robots-txt-rules)
- [Shopify: Customize robots.txt](https://shopify.dev/docs/storefronts/themes/seo/robots-txt)
- [Next.js: robots.txt file convention](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots)

## FAQ

### Will blocking AI crawlers hurt my Google rankings?

Blocking Google-Extended doesn't affect Google Search. Blocking Googlebot does, and it also removes you from AI Overviews and AI Mode. Blocking OpenAI, Anthropic or Perplexity crawlers only affects those products.

### Do AI crawlers respect robots.txt?

OpenAI, Anthropic and Perplexity document that their crawlers follow robots.txt. User-triggered fetchers differ: OpenAI says robots.txt may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores it, while Anthropic says its bots, including Claude-User, honor it.

### Can I block AI training but stay in ChatGPT search?

Yes. Disallow GPTBot and allow OAI-SearchBot. OpenAI controls them separately, and says changes take about 24 hours to apply to search.

### My robots.txt allows AI bots. Why am I still blocked?

Usually a CDN or firewall rule. On Cloudflare, check the AI bot policies under Security Settings: since September 15, 2026, new domains block Training and Agent bots on pages with ads by default. On Vercel, check whether the AI Bots managed ruleset is set to Deny. The [AI crawler checker](https://getrankroot.com/tools/ai-crawler-checker) shows which bots actually get through.

### Does Cloudflare's managed robots.txt replace my file?

No. Cloudflare says that if your site already serves a robots.txt (an HTTP 200 response), Bot Preference Sync prepends its managed rules before your content. If you have none, it creates one. It only adds rules when at least one AI bot policy blocks or disallows; with every policy on Allow, your file is served unchanged. Its managed rules currently add use=reference to the Content-Signal line, an extension Cloudflare says it is testing.

## Check your site

Run the free audit at https://getrankroot.com/ or call `GET https://getrankroot.com/api/audit?url=example.com`.

## More guides

- [How to get your product recommended by ChatGPT, Perplexity and Claude](https://getrankroot.com/guides/how-to-get-recommended-by-chatgpt.md)
- [What is llms.txt? Format, examples and a free generator](https://getrankroot.com/guides/what-is-llms-txt.md)
- [Schema markup for AI search: the JSON-LD that helps AI describe you](https://getrankroot.com/guides/schema-markup-for-ai-search.md)
- [How to make your SaaS agent-ready (MCP, OpenAPI, llms.txt)](https://getrankroot.com/guides/make-your-saas-agent-ready.md)
- [How to track traffic from ChatGPT, Perplexity and Claude in GA4](https://getrankroot.com/guides/track-ai-traffic-ga4.md)
- [GEO vs SEO: what generative engine optimization actually changes](https://getrankroot.com/guides/geo-vs-seo.md)
- [How to check if ChatGPT mentions your brand (manual method + tools)](https://getrankroot.com/guides/check-if-chatgpt-mentions-your-brand.md)
- [llms.txt for WordPress: how to add it (manual, Yoast, plugins)](https://getrankroot.com/guides/llms-txt-wordpress.md)
- [Should you block GPTBot? AI training vs search crawlers explained](https://getrankroot.com/guides/should-you-block-gptbot.md)
- [Schema markup for SaaS: the JSON-LD worth adding for AI search](https://getrankroot.com/guides/schema-markup-for-saas.md)
- [GPTBot vs OAI-SearchBot vs ChatGPT-User: what to block](https://getrankroot.com/guides/gptbot-vs-oai-searchbot-vs-chatgpt-user.md)
- [ClaudeBot vs Claude-SearchBot vs Claude-User explained](https://getrankroot.com/guides/claudebot-vs-claude-searchbot-vs-claude-user.md)
- [What is an AI visibility score? (and how to check yours)](https://getrankroot.com/guides/what-is-an-ai-visibility-score.md)
