AI crawler list · Google · Control token (does not crawl)
Google-Extended
A robots.txt token, not a crawler. It controls whether content Google already crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google says it does not affect inclusion in Google Search and is not a ranking signal.
Facts checked against Google docs on Oct 20, 2026.
Is Google-Extended allowed on your site?
Reads your robots.txt and requests your page as each AI bot. Free, no signup.
Should you allow it?
Your choice. Blocking it does not remove you from Google Search.
If you block it: Google won't use your content to train future Gemini models or for grounding in Gemini Apps and Vertex AI. Google Search is unaffected.
Google crawls with its normal user agents; Google-Extended only governs how that content is used.
AI Overviews and AI Mode are part of Google Search. Google's controls for them are Googlebot's robots.txt rules and the nosnippet, data-nosnippet, max-snippet and noindex rules, not Google-Extended.
robots.txt rules
User-agent: Google-Extended Allow: /
User-agent: Google-Extended Disallow: /
User-agent: Google-Extended Disallow: /private/ Allow: /
Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.
Same company, different bots: Google
Google uses separate user agents for separate jobs. Blocking one does not block the others.
| Bot | Purpose | Obeys robots.txt | If you block it |
|---|---|---|---|
Google-ExtendedControl token (does not crawl) | Gemini training and grounding | Token only | Google won't use your content to train future Gemini models or for grounding in Gemini Apps and Vertex AI. Google Search is unaffected. |
GooglebotAI search index | Google Search (incl. AI Overviews) | Yes | Google can't crawl your pages for Search, Discover, Images, Video or News, and they can't be shown with content in AI Overviews or AI Mode. A blocked URL can still be listed without a description if other sites link to it. |
Google-CloudVertexBotUser-triggered fetcher | Vertex AI Agents | Yes | Vertex AI agents that site owners request can't crawl your site. No effect on Google Search. |
User-agent: Googlebot User-agent: Google-CloudVertexBot Allow: / User-agent: Google-Extended Disallow: /
How to verify it's really Google-Extended
Google-Extended never sends requests, so it never appears in your server logs. The traffic it governs comes from Googlebot; verify that crawler instead.
Common mistakes
- Creating a firewall or user agent rule for Google-Extended. Google says it has no separate user agent string, so no request will ever match it.
- Disallowing it to get out of AI Overviews. Google points to Googlebot's robots.txt rules and snippet controls for Search features; Google-Extended governs Gemini training and grounding.
FAQ
A robots.txt token, not a crawler. It controls whether content Google already crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google says it does not affect inclusion in Google Search and is not a ranking signal.
Google-Extended is a robots.txt token only; it never visits your site. A robots.txt token, not a crawler. It controls whether content Google already crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google says it does not affect inclusion in Google Search and is not a ranking signal.
Google won't use your content to train future Gemini models or for grounding in Gemini Apps and Vertex AI. Google Search is unaffected.
Your choice. Blocking it does not remove you from Google Search.
No. Google says crawling is done with existing Google user agent strings; Google-Extended is only a robots.txt control token. It never appears in your server logs.
Google's documentation does not say so. AI Overviews and AI Mode are Search features; Google says Googlebot's robots.txt rules are the control for crawling for Search, and nosnippet, data-nosnippet, max-snippet or noindex limit what Search shows. Google-Extended covers Gemini training and grounding.
No. Google says Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal.
Sources
- Official documentation from Google
- Google: how to verify its crawlers
- Google: AI features and your website
Related AI crawlers
Googlebot · Google-CloudVertexBot · Applebot-Extended
Facts checked against Google docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me