# Cloudflare Pages & Workers SEO for ChatGPT, Perplexity and AI search

> Get your Cloudflare Pages or Workers site into ChatGPT and AI search: AI bot policies, managed robots.txt, Markdown for Agents, _headers and noindex previews.

Canonical page: https://getrankroot.com/builders/cloudflare · Web hosting platform · checked 2026-10-22

What Cloudflare Pages & Workers's own documentation says about how AI crawlers see sites built with it, and the exact settings for robots.txt, llms.txt, structured data and the sitemap.

## How Cloudflare Pages & Workers delivers pages to crawlers

**Depends on what you deploy** (Depends on your project)

Cloudflare Pages and Workers static assets serve the files your build outputs, and framework adapters can render pages on the server, so static and server-rendered sites send crawlers real HTML. A client-side single-page app is different: with Pages, if there's no top-level `404.html`, Cloudflare assumes an SPA and serves the root page for every unknown path; with Workers, `not_found_handling: "single-page-application"` serves `/index.html` with a 200. Either way crawlers that don't run JavaScript get the same near-empty shell on every URL. We didn't find a Cloudflare feature that pre-renders client-only apps for crawlers, so use your framework's static export or server rendering. Run RankRoot's audit to see what crawlers receive.

Why it matters: many AI crawlers don't run JavaScript. If your text only appears after scripts run, they may see an empty page.

## Make your content readable without JavaScript

1. Check the AI bot policies: in the dashboard open **Security Settings**, filter by **Bot traffic** and choose **Configure AI bot policies**. Search, Agent and Training bots each have **Block**, **Block on pages with ads** or **Allow**. Cloudflare says that from September 15, 2026, new domains block Training and Agent bots on pages with ads, while Search stays allowed. Agent covers chat fetch bots acting for a person, so set it to Allow if you want AI assistants to read your pages live.
2. The older **Block AI bots** setting is marked as deprecating on September 15, 2026. Cloudflare says crawlers used for both Search and Training are blocked by every configuration that blocks AI training.
3. Connect a custom domain. Cloudflare recommends running production Workers on a route or custom domain, and describes workers.dev as a Free-plan subdomain for personal or hobby projects.
4. Run the [free audit](https://getrankroot.com/#scan) and the [AI crawler checker](https://getrankroot.com/tools/ai-crawler-checker) on your custom domain.

## robots.txt: let AI search crawlers in

1. Put `robots.txt` in your static assets folder (`public/` or `static/` in most frameworks) so it lands in the build output, or generate it with your framework (for Next.js, `app/robots.ts`).
2. If you turn on **Enable Bot Preference Sync** in the AI bot policies (Cloudflare's managed robots.txt, available on all plans), Cloudflare prepends its managed rules before your existing robots.txt, or creates one if you have none. Open `yourdomain.com/robots.txt` afterwards to see the combined file.
3. robots.txt can't override a block at Cloudflare's edge. If a bot policy blocks a crawler, it stays blocked whatever robots.txt allows.

robots.txt that lets AI search crawlers in:

```
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml
```

## Add llms.txt

1. Cloudflare has no llms.txt setting. Save the file as `llms.txt` in your static assets folder, next to `robots.txt`, and it's served at `/llms.txt`.
2. Write it with the [llms.txt generator](https://getrankroot.com/tools/llms-txt-generator), deploy, then check the live file with the [llms.txt validator](https://getrankroot.com/tools/llms-txt-validator).
3. On a Pro or Business plan, **Markdown for Agents** (zone > **AI Crawl Control**) converts your HTML pages to Markdown when an AI client sends `Accept: text/markdown`. Cloudflare says it's free on those plans, only converts HTML, and adds a default `Content-Signal: ai-train=yes, search=yes, ai-input=yes` header unless your origin sets its own.

## Add JSON-LD and head code

1. Set titles, meta descriptions, canonical tags and JSON-LD in your framework's layout or head component.
2. Use a `_headers` file in your static assets folder for response headers on static files (up to 100 rules, 2,000 characters per line). Cloudflare says `_headers` is not applied to responses from Pages Functions or Worker code; set those headers in the code.
3. Keep `*.pages.dev` and `*.workers.dev` out of search with an `X-Robots-Tag: noindex` rule for those hostnames, and a canonical tag pointing to your custom domain. Cloudflare's own `_headers` docs show this rule.

_headers: noindex on pages.dev (Pages) or workers.dev (Workers) only:

```
# Cloudflare Pages
https://:project.pages.dev/*
  X-Robots-Tag: noindex

https://:version.:project.pages.dev/*
  X-Robots-Tag: noindex

# Workers static assets (use your own Worker and subdomain)
https://my-worker.my-subdomain.workers.dev/*
  X-Robots-Tag: noindex
```

## Sitemap

1. Cloudflare doesn't generate a sitemap. Use your framework's option (for Next.js, `app/sitemap.ts`) or a static `sitemap.xml` in the static assets folder, listing URLs on your custom domain.
2. Add a `Sitemap:` line to `robots.txt` and submit it in Google Search Console and Bing Webmaster Tools.

## Common mistakes

- Leaving the default AI bot policy in place on a domain that shows ads. Since September 15, 2026, new domains block Agent bots on pages with ads, so ChatGPT and Claude can't fetch those pages live for a user.
- Setting Training to Block and expecting search citations to be untouched. Cloudflare says crawlers used for both Search and Training are blocked by every configuration that blocks training.
- Shipping a client-side SPA. Every URL returns the same shell, and AI crawlers that don't run JavaScript see no text.
- Adding headers in `_headers` and expecting them on SSR or Function routes. Cloudflare doesn't apply `_headers` to responses generated by your code.
- Disabling the workers.dev route only in the dashboard. Cloudflare says Wrangler re-enables it on the next deploy unless you set `"workers_dev": false` in your config.
- Treating preview URLs as private. Pages previews and workers.dev Preview URLs are public by default and send `X-Robots-Tag: noindex`; the docs only promise that header on workers.dev, and recommend Cloudflare Access for Previews on a custom domain.

## FAQ

### Does Cloudflare block ChatGPT and other AI crawlers?

It depends on your AI bot policies. Cloudflare says new domains since September 15, 2026 block Training and Agent bots on pages with ads and allow Search. Check Security Settings > Configure AI bot policies and set Search and Agent to Allow if you want AI answers to cite and read your pages.

### Where does robots.txt go on Cloudflare Pages or Workers?

In your static assets folder (public/ or static/ in most frameworks), so it ends up in the build output. If Bot Preference Sync is on, Cloudflare prepends its managed rules to your file.

### Are Cloudflare Pages preview deployments indexed?

Cloudflare says every Pages preview deployment sends X-Robots-Tag: noindex, and workers.dev Preview URLs do too. Your production pages.dev or workers.dev URL doesn't, so add a noindex rule for it in _headers and a canonical tag to your domain.

### What is Cloudflare Markdown for Agents?

A free feature on Pro, Business and Enterprise plans that converts HTML to Markdown at Cloudflare's edge when an AI client asks for text/markdown. Turn it on in the zone's AI Crawl Control section.

## Sources

- [Cloudflare: Block AI bots and AI bot policies (read 2026-10-22)](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/)
- [Cloudflare: Managed robots.txt (Bot Preference Sync) (read 2026-10-22)](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/)
- [Cloudflare: Markdown for Agents (read 2026-10-22)](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/)
- [Cloudflare Pages: Serving Pages (SPA rendering) (read 2026-10-22)](https://developers.cloudflare.com/pages/configuration/serving-pages/)
- [Cloudflare Pages: Preview deployments (read 2026-10-22)](https://developers.cloudflare.com/pages/configuration/preview-deployments/)
- [Cloudflare Pages: Headers (read 2026-10-22)](https://developers.cloudflare.com/pages/configuration/headers/)
- [Cloudflare Workers: Static assets headers (read 2026-10-22)](https://developers.cloudflare.com/workers/static-assets/headers/)
- [Cloudflare Workers: Single Page Application routing (read 2026-10-22)](https://developers.cloudflare.com/workers/static-assets/routing/single-page-application/)
- [Cloudflare Workers: Previews (read 2026-10-22)](https://developers.cloudflare.com/workers/previews/)
- [Cloudflare Workers: workers.dev routing (read 2026-10-22)](https://developers.cloudflare.com/workers/configuration/routing/workers-dev/)
- [Next.js: robots.txt file convention (read 2026-10-18)](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots)

RankRoot is not affiliated with Cloudflare Pages & Workers. Check a live site: https://getrankroot.com/ or `GET https://getrankroot.com/api/audit?url=example.com`. Other builders: https://getrankroot.com/builders.md