Website & app builders · Web hosting platform · Checked 2026-10-22

Cloudflare Pages & Workers SEO for ChatGPT, Perplexity & AI search

Built your site with Cloudflare Pages & Workers? Here is what Cloudflare Pages & Workers's own documentation says about how AI crawlers see it, and the exact settings for robots.txt, llms.txt, structured data and your sitemap. Sources are listed at the bottom.

Is your Cloudflare Pages & Workers site visible to AI? Check it free

Paste your live URL. RankRoot checks your robots.txt rules for the ChatGPT, Perplexity and Claude crawlers, whether your page text is in the HTML without JavaScript, and your llms.txt and JSON-LD.

Free, no signup. 20+ checks across ChatGPT, Perplexity, Claude, Google AI and AI agents.

How Cloudflare Pages & Workers delivers pages to crawlers

Rendering

Depends on what you deploy

Depends on your project

Cloudflare Pages and Workers static assets serve the files your build outputs, and framework adapters can render pages on the server, so static and server-rendered sites send crawlers real HTML. A client-side single-page app is different: with Pages, if there's no top-level 404.html, Cloudflare assumes an SPA and serves the root page for every unknown path; with Workers, not_found_handling: "single-page-application" serves /index.html with a 200. Either way crawlers that don't run JavaScript get the same near-empty shell on every URL. We didn't find a Cloudflare feature that pre-renders client-only apps for crawlers, so use your framework's static export or server rendering. Run RankRoot's audit to see what crawlers receive.

Why it matters: many AI crawlers don't run JavaScript. If your text only appears after scripts run, they may see an empty page.

Make your content readable without JavaScript

Check the AI bot policies: in the dashboard open Security Settings, filter by Bot traffic and choose Configure AI bot policies. Search, Agent and Training bots each have Block, Block on pages with ads or Allow. Cloudflare says that from September 15, 2026, new domains block Training and Agent bots on pages with ads, while Search stays allowed. Agent covers chat fetch bots acting for a person, so set it to Allow if you want AI assistants to read your pages live.
The older Block AI bots setting is marked as deprecating on September 15, 2026. Cloudflare says crawlers used for both Search and Training are blocked by every configuration that blocks AI training.
Connect a custom domain. Cloudflare recommends running production Workers on a route or custom domain, and describes workers.dev as a Free-plan subdomain for personal or hobby projects.
Run the free audit and the AI crawler checker on your custom domain.

robots.txt: let AI search crawlers in

Put robots.txt in your static assets folder (public/ or static/ in most frameworks) so it lands in the build output, or generate it with your framework (for Next.js, app/robots.ts).
If you turn on Enable Bot Preference Sync in the AI bot policies (Cloudflare's managed robots.txt, available on all plans), Cloudflare prepends its managed rules before your existing robots.txt, or creates one if you have none. Open yourdomain.com/robots.txt afterwards to see the combined file.
robots.txt can't override a block at Cloudflare's edge. If a bot policy blocks a crawler, it stays blocked whatever robots.txt allows.

robots.txt that lets AI search crawlers in

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Which bots to allow or block: robots.txt for AI crawlers and the AI crawler list.

Add llms.txt

Cloudflare has no llms.txt setting. Save the file as llms.txt in your static assets folder, next to robots.txt, and it's served at /llms.txt.
Write it with the llms.txt generator, deploy, then check the live file with the llms.txt validator.
On a Pro or Business plan, Markdown for Agents (zone > AI Crawl Control) converts your HTML pages to Markdown when an AI client sends Accept: text/markdown. Cloudflare says it's free on those plans, only converts HTML, and adds a default Content-Signal: ai-train=yes, search=yes, ai-input=yes header unless your origin sets its own.

Add JSON-LD and head code

Set titles, meta descriptions, canonical tags and JSON-LD in your framework's layout or head component.
Use a _headers file in your static assets folder for response headers on static files (up to 100 rules, 2,000 characters per line). Cloudflare says _headers is not applied to responses from Pages Functions or Worker code; set those headers in the code.
Keep *.pages.dev and *.workers.dev out of search with an X-Robots-Tag: noindex rule for those hostnames, and a canonical tag pointing to your custom domain. Cloudflare's own _headers docs show this rule.

_headers: noindex on pages.dev (Pages) or workers.dev (Workers) only

# Cloudflare Pages
https://:project.pages.dev/*
  X-Robots-Tag: noindex

https://:version.:project.pages.dev/*
  X-Robots-Tag: noindex

# Workers static assets (use your own Worker and subdomain)
https://my-worker.my-subdomain.workers.dev/*
  X-Robots-Tag: noindex

Sitemap

Cloudflare doesn't generate a sitemap. Use your framework's option (for Next.js, app/sitemap.ts) or a static sitemap.xml in the static assets folder, listing URLs on your custom domain.
Add a Sitemap: line to robots.txt and submit it in Google Search Console and Bing Webmaster Tools.

Common mistakes

Leaving the default AI bot policy in place on a domain that shows ads. Since September 15, 2026, new domains block Agent bots on pages with ads, so ChatGPT and Claude can't fetch those pages live for a user.
Setting Training to Block and expecting search citations to be untouched. Cloudflare says crawlers used for both Search and Training are blocked by every configuration that blocks training.
Shipping a client-side SPA. Every URL returns the same shell, and AI crawlers that don't run JavaScript see no text.
Adding headers in _headers and expecting them on SSR or Function routes. Cloudflare doesn't apply _headers to responses generated by your code.
Disabling the workers.dev route only in the dashboard. Cloudflare says Wrangler re-enables it on the next deploy unless you set "workers_dev": false in your config.
Treating preview URLs as private. Pages previews and workers.dev Preview URLs are public by default and send X-Robots-Tag: noindex; the docs only promise that header on workers.dev, and recommend Cloudflare Access for Previews on a custom domain.

Get the files for your Cloudflare Pages & Workers site

Generate an llms.txt, then validate the live file once it's published.

llms.txt generator Validate llms.txt

Rather have this done for you?

The AI Visibility Employee looks after AI visibility for your Cloudflare Pages & Workers site every month, fully by email, no calls:

Weekly tracking of how ChatGPT, Perplexity, Gemini and Claude mention you for your buyers' questions
4 answer-ready pages a month, published through our WordPress plugin or a GitHub pull request, or delivered as drafts you add in Cloudflare Pages & Workers
Ongoing llms.txt, robots.txt and JSON-LD fixes wherever your setup allows them, and a monthly report

$149/month, cancel anytime. The first 10 customers pay $99/month with code FOUNDING10.

See the AI Visibility Employee

FAQ

Does Cloudflare block ChatGPT and other AI crawlers?

It depends on your AI bot policies. Cloudflare says new domains since September 15, 2026 block Training and Agent bots on pages with ads and allow Search. Check Security Settings > Configure AI bot policies and set Search and Agent to Allow if you want AI answers to cite and read your pages.

Where does robots.txt go on Cloudflare Pages or Workers?

In your static assets folder (public/ or static/ in most frameworks), so it ends up in the build output. If Bot Preference Sync is on, Cloudflare prepends its managed rules to your file.

Are Cloudflare Pages preview deployments indexed?

Cloudflare says every Pages preview deployment sends X-Robots-Tag: noindex, and workers.dev Preview URLs do too. Your production pages.dev or workers.dev URL doesn't, so add a noindex rule for it in _headers and a canonical tag to your domain.

What is Cloudflare Markdown for Agents?

A free feature on Pro, Business and Enterprise plans that converts HTML to Markdown at Cloudflare's edge when an AI client asks for text/markdown. Turn it on in the zone's AI Crawl Control section.

Sources

Facts checked against Cloudflare Pages & Workers's documentation on 2026-10-22. RankRoot is not affiliated with Cloudflare Pages & Workers. Settings move around; if a step has changed, email rankroot.support@proton.me.

Other builders

Lovable · Bolt.new · Replit · v0 · Framer · Webflow · Wix · Squarespace · Shopify · WordPress · Ghost · GitHub Pages · Carrd · Hostinger AI Builder · Notion Sites · Netlify · Vercel