# GitHub Pages SEO for ChatGPT, Perplexity and AI search

> Get your GitHub Pages site into ChatGPT and AI search: where robots.txt and llms.txt go, jekyll-seo-tag and jekyll-sitemap, and project site limits.

Canonical page: https://getrankroot.com/builders/github-pages · Static site hosting · checked 2026-10-19

What GitHub Pages's own documentation says about how AI crawlers see sites built with it, and the exact settings for robots.txt, llms.txt, structured data and the sitemap.

## How GitHub Pages delivers pages to crawlers

**Static files, served as published** (Depends on your project)

GitHub's docs describe Pages as a static site hosting service that takes HTML, CSS and JavaScript files from a repository, optionally running them through a build (Jekyll by default, or any generator through GitHub Actions). Crawlers get exactly the files you publish. A Jekyll, Hugo or Astro site puts the text in the HTML. A single-page app built with React or Vite usually ships a near-empty HTML file and fills it in with JavaScript, which many AI crawlers don't run. Run RankRoot's audit to see which one you have.

Why it matters: many AI crawlers don't run JavaScript. If your text only appears after scripts run, they may see an empty page.

## Make your content readable without JavaScript

1. Use a custom domain if you can. A project site lives at `<owner>.github.io/<repository>`, and crawlers only read `robots.txt` from the root of a host, so a project site can't set its own crawler rules on github.io.
2. If your site is a client-side app, pre-render it: use your framework's static export or a static site generator so each page's text is in the HTML file.
3. Run the [free audit](https://getrankroot.com/#scan) on your live URL to confirm crawlers see your text without JavaScript.

## robots.txt: let AI search crawlers in

1. Add a file named `robots.txt` to the root of what you publish: the repository root or `/docs` folder if you deploy from a branch, or your build's output folder if you deploy with GitHub Actions.
2. With Jekyll, a file without front matter is a static file and is copied to the site unchanged, so a plain `robots.txt` works as is.
3. For a project site without a custom domain, `robots.txt` has to live in the `<owner>.github.io` user or organization site, because crawlers only check `/robots.txt` at the top of the host.
4. Push, then open `yourdomain.com/robots.txt` to confirm the live file.

robots.txt that lets AI search crawlers in:

```
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml
```

## Add llms.txt

1. GitHub Pages has no llms.txt feature, but any file you publish is served. Write the file with the [llms.txt generator](https://getrankroot.com/tools/llms-txt-generator) and save it as `llms.txt` next to `robots.txt`.
2. Leave front matter out so Jekyll copies it unchanged. Jekyll skips files and folders whose names start with `_`, `.` or `#`, so don't put it in one of those.
3. Push, open `yourdomain.com/llms.txt` and check it with the [llms.txt validator](https://getrankroot.com/tools/llms-txt-validator).

## Add JSON-LD and head code

1. On Jekyll, add the `jekyll-seo-tag` plugin. It's on GitHub Pages' list of supported plugins. List it under `plugins` in `_config.yml` and put `{% seo %}` right before `</head>` in your layout.
2. jekyll-seo-tag outputs the page title, description, canonical URL, Open Graph and Twitter card tags, and JSON-LD for site and post metadata. Set `title` and `description` in `_config.yml` and in each page's front matter.
3. Not using Jekyll? Add the title, meta description and a `<script type="application/ld+json">` block in your generator's layout or head component.

_config.yml for jekyll-seo-tag and jekyll-sitemap:

```
url: "https://yourdomain.com"
title: Your brand
description: One sentence on what you do and who it's for.
plugins:
  - jekyll-seo-tag
  - jekyll-sitemap
```

## Sitemap

1. On Jekyll, add `jekyll-sitemap` (also on GitHub Pages' supported list) under `plugins` in `_config.yml`, and set `url` to your site's address. It generates `sitemap.xml`.
2. To leave a page out, set `sitemap: false` in its front matter.
3. With other generators, use the generator's sitemap option and deploy the output with GitHub Actions. Add a `Sitemap:` line to `robots.txt` and submit the sitemap in Google Search Console.

## Common mistakes

- Deploying a React or Vite single-page app and assuming AI crawlers see the content. Without pre-rendering, they often get an empty `<div id="root">`.
- Adding `robots.txt` to a project site at `<owner>.github.io/<repository>/robots.txt`. Crawlers don't look there.
- Adding plugins to the Gemfile only. The jekyll-sitemap docs warn that the GitHub Pages gem ignores plugins that aren't listed in `_config.yml`.
- Forgetting `url` in `_config.yml`, so the sitemap and canonical tags point to the wrong host.

## FAQ

### Can I add llms.txt to GitHub Pages?

Yes. Save a plain llms.txt file (no front matter) at the root of what you publish and push. GitHub Pages serves it like any other static file.

### Does GitHub Pages create a sitemap?

Not on its own. On Jekyll, add the jekyll-sitemap plugin in _config.yml (GitHub Pages supports it). With other generators, use their sitemap option.

### Is my GitHub Pages site visible to ChatGPT?

If your pages are pre-built HTML (Jekyll, Hugo, Astro and similar), yes, as long as robots.txt allows OAI-SearchBot. A client-side single-page app may look empty to crawlers that don't run JavaScript. Run the free audit to check.

### Where does robots.txt go on GitHub Pages?

At the root of your host. For a custom domain or a <owner>.github.io site that's the root of your published files. A project site on github.io can't have its own robots.txt.

## Sources

- [GitHub Docs: What is GitHub Pages? (read 2026-10-19)](https://docs.github.com/en/pages/getting-started-with-github-pages/what-is-github-pages)
- [GitHub Docs: About GitHub Pages and Jekyll (read 2026-10-19)](https://docs.github.com/en/pages/setting-up-a-github-pages-site-with-jekyll/about-github-pages-and-jekyll)
- [GitHub Docs: Configuring a publishing source (read 2026-10-19)](https://docs.github.com/en/pages/getting-started-with-github-pages/configuring-a-publishing-source-for-your-github-pages-site)
- [GitHub Docs: Using custom workflows with GitHub Pages (read 2026-10-19)](https://docs.github.com/en/pages/getting-started-with-github-pages/using-custom-workflows-with-github-pages)
- [GitHub Docs: About custom domains and GitHub Pages (read 2026-10-19)](https://docs.github.com/en/pages/configuring-a-custom-domain-for-your-github-pages-site/about-custom-domains-and-github-pages)
- [GitHub Pages: Dependency versions (read 2026-10-19)](https://pages.github.com/versions)
- [jekyll-seo-tag: Installation (read 2026-10-19)](https://github.com/jekyll/jekyll-seo-tag/blob/master/docs/installation.md)
- [jekyll-sitemap README (read 2026-10-19)](https://github.com/jekyll/jekyll-sitemap)
- [Jekyll docs: Static files (read 2026-10-19)](https://jekyllrb.com/docs/static-files/)
- [RFC 9309: Robots Exclusion Protocol (read 2026-10-19)](https://www.rfc-editor.org/rfc/rfc9309.html)

RankRoot is not affiliated with GitHub Pages. Check a live site: https://getrankroot.com/ or `GET https://getrankroot.com/api/audit?url=example.com`. Other builders: https://getrankroot.com/builders.md