Website & app builders · Web hosting platform · Checked 2026-10-21

Vercel SEO for ChatGPT, Perplexity & AI search

Built your site with Vercel? Here is what Vercel's own documentation says about how AI crawlers see it, and the exact settings for robots.txt, llms.txt, structured data and your sitemap. Sources are listed at the bottom.

How Vercel delivers pages to crawlers

Vercel runs your framework's build: Next.js, SvelteKit, Astro and similar frameworks render pages to HTML on the server or at build time, so crawlers get your text. A client-only React or Vite app ships a near-empty HTML file that many AI crawlers can't fill in, because they don't run JavaScript. We didn't find a Vercel feature that pre-renders client-only apps for crawlers, so use your framework's static export or server rendering. Run RankRoot's audit to see what crawlers receive.

Why it matters: many AI crawlers don't run JavaScript. If your text only appears after scripts run, they may see an empty page.

Make your content readable without JavaScript

1. Check the firewall: project Firewall > Rules > Bot Management. The AI Bots Ruleset is off by default (labeled Allow). Set to Deny, it blocks all traffic Vercel identifies as AI bots, which Vercel says covers bots that crawl for training, search and user-requested fetches. Use Log if you only want to watch.
2. The Bot Protection ruleset is also off by default. In challenge mode it serves a JavaScript challenge to traffic unlikely to be a browser, but Vercel says it skips verified bots. Vercel's verified bots directory lists OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Claude-User.
3. Check Settings > Deployment Protection. Since September 2026, Vercel Authentication can protect production too, on every plan. If it covers the URL you want cited, visitors must sign in with Vercel, and crawlers can't.
4. Run the free audit and the AI crawler checker on your custom domain.

robots.txt: let AI search crawlers in

1. Vercel serves the robots.txt your framework outputs. On Next.js add app/robots.ts or public/robots.txt; on Vite, Astro or SvelteKit put robots.txt in the static folder.
2. Deploy and open yourdomain.com/robots.txt. A firewall rule can still block bots that robots.txt allows, so check the AI Bots ruleset too.

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Which bots to allow or block: robots.txt for AI crawlers and the AI crawler list.

Add llms.txt

1. Vercel has no llms.txt setting. Save the file in your framework's static folder (public/llms.txt in Next.js) and it's served at /llms.txt.
2. Write it with the llms.txt generator, deploy, then check yourdomain.com/llms.txt with the llms.txt validator.

Add JSON-LD and head code

1. Set the title, meta description and JSON-LD in your framework. In Next.js use the Metadata API and render a <script type="application/ld+json"> tag in your layout or page.
2. Add a canonical tag on every page that points to your custom domain. Vercel recommends it so your-project.vercel.app doesn't compete with your domain.
3. For extra safety, send X-Robots-Tag: noindex only on the vercel.app host. Vercel prefers setting it in your framework config and using vercel.json only when the framework can't set headers.

{
  "headers": [
    {
      "source": "/(.*)",
      "has": [{ "type": "host", "value": "your-project.vercel.app" }],
      "headers": [{ "key": "X-Robots-Tag", "value": "noindex" }]
    }
  ]
}

Sitemap

1. Vercel doesn't generate a sitemap. Use your framework's option (for Next.js, app/sitemap.ts) and list URLs on your custom domain, not vercel.app.
2. Reference it from robots.txt with a Sitemap: line and submit it in Google Search Console.

Common mistakes

• Setting the AI Bots ruleset to Deny to stop training crawlers. It blocks AI search and user-requested fetches too. To block training only, disallow GPTBot or ClaudeBot in robots.txt and leave the ruleset on Allow or Log.
• Turning on Deployment Protection for all deployments, including production, on a public marketing site. Crawlers can't sign in.
• Assigning a custom domain to a preview branch. Vercel then leaves out the automatic X-Robots-Tag: noindex, so that domain can be indexed as duplicate content.
• Leaving your-project.vercel.app indexable next to your domain. Add canonical tags and, if needed, a noindex header for that host.

FAQ

Sources

• Vercel Docs: Bot Management (AI bots ruleset, verified bots) (read 2026-10-21)
• Vercel Docs: WAF Managed Rulesets (read 2026-10-21)
• Vercel Docs: Response headers (x-robots-tag) (read 2026-10-21)
• Vercel KB: Are Vercel Preview Deployments indexed by search engines? (read 2026-10-21)
• Vercel KB: Avoiding duplicate content with vercel.app URLs (read 2026-10-21)
• Vercel Docs: Vercel Authentication (read 2026-10-21)
• Vercel changelog: Protect production deployments for free on every plan (Sept 9, 2026) (read 2026-10-21)
• Next.js: robots.txt file convention (read 2026-10-18)
• Next.js: sitemap.xml file convention (read 2026-10-18)
• Next.js: JSON-LD (read 2026-10-18)

Facts checked against Vercel's documentation on 2026-10-21. RankRoot is not affiliated with Vercel. Settings move around; if a step has changed, email rankroot.support@proton.me.

Other builders

Lovable · Bolt.new · Replit · v0 · Framer · Webflow · Wix · Squarespace · Shopify · WordPress · Ghost · GitHub Pages · Carrd · Hostinger AI Builder · Notion Sites · Netlify · Cloudflare Pages & Workers