AI crawler list · Google · AI search index
Googlebot
Google's main search crawler. Google says Googlebot's robots.txt rules are the control for crawling for Search, and its AI features (AI Overviews and AI Mode) only link to pages that are indexed and eligible to show with a snippet.
Facts checked against Google docs on Oct 20, 2026.
Is Googlebot allowed on your site?
Reads your robots.txt and requests your page as each AI bot. Free, no signup.
Should you allow it?
Always allow it.
If you block it: Google can't crawl your pages for Search, Discover, Images, Video or News, and they can't be shown with content in AI Overviews or AI Mode. A blocked URL can still be listed without a description if other sites link to it.
The first string is Googlebot Smartphone, then Googlebot Desktop; Google says the last two are used rarely. W.X.Y.Z is the Chrome version, which changes over time, so match it with a wildcard.
robots.txt rules
User-agent: Googlebot Allow: /
User-agent: Googlebot Disallow: /
User-agent: Googlebot Disallow: /private/ Allow: /
Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.
Same company, different bots: Google
Google uses separate user agents for separate jobs. Blocking one does not block the others.
| Bot | Purpose | Obeys robots.txt | If you block it |
|---|---|---|---|
Google-ExtendedControl token (does not crawl) | Gemini training and grounding | Token only | Google won't use your content to train future Gemini models or for grounding in Gemini Apps and Vertex AI. Google Search is unaffected. |
GooglebotAI search index | Google Search (incl. AI Overviews) | Yes | Google can't crawl your pages for Search, Discover, Images, Video or News, and they can't be shown with content in AI Overviews or AI Mode. A blocked URL can still be listed without a description if other sites link to it. |
Google-CloudVertexBotUser-triggered fetcher | Vertex AI Agents | Yes | Vertex AI agents that site owners request can't crawl your site. No effect on Google Search. |
User-agent: Googlebot User-agent: Google-CloudVertexBot Allow: / User-agent: Google-Extended Disallow: /
User agent string
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html)
Versions in the string can change. Match on the token Googlebot, and verify real traffic as shown below rather than trusting the header.
How to verify it's really Googlebot
Find its requests in your server logs (the path to your access log depends on your host):
grep -i "Googlebot" access.log
Google's common crawlers such as Googlebot resolve to hostnames like crawl-***.googlebot.com or geo-crawl-***.geo.googlebot.com.
Reverse DNS check (Google documents this): look up the IP from your log, confirm the hostname ends in googlebot.com, google.com or googleusercontent.com, then look up that hostname and confirm it returns the same IP.
host 203.0.113.7 # replace with the IP from your log host <hostname-from-above> # must return the same IP
Published IP list: developers.google.com/static/crawling/ipranges/common-crawlers.json
Check one IP against the list (prints True or False):
curl -s https://developers.google.com/static/crawling/ipranges/common-crawlers.json | python3 -c "import sys,re,ipaddress as a; ip=a.ip_address(sys.argv[1]); print(any(ip in a.ip_network(c,strict=False) for c in re.findall(r'[0-9a-fA-F:.]+/[0-9]+', sys.stdin.read())))" 203.0.113.7
Common mistakes
- Using Disallow to hide a page from Google. Google says robots.txt is not a way to keep a page out of Google: a disallowed URL can still be indexed if other sites link to it. Use noindex instead.
- Combining noindex with a robots.txt Disallow. Google says the page must not be blocked by robots.txt, or Googlebot never sees the noindex rule.
- Setting Crawl-delay for Googlebot. Google supports only user-agent, allow, disallow and sitemap; crawl-delay is ignored.
- Writing a User-agent: Googlebot group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the Googlebot group.
- Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
- Trusting the user agent header alone. Anyone can send "Googlebot" in a request; verify the source IP as shown above before allowlisting or blocking.
FAQ
Google's main search crawler. Google says Googlebot's robots.txt rules are the control for crawling for Search, and its AI features (AI Overviews and AI Mode) only link to pages that are indexed and eligible to show with a snippet.
Yes, according to Google.
Google can't crawl your pages for Search, Discover, Images, Video or News, and they can't be shown with content in AI Overviews or AI Mode. A blocked URL can still be listed without a description if other sites link to it.
Always allow it.
Effectively yes, along with the rest of Google Search. Google says Googlebot's robots.txt rules are the control for crawling for Search, and a page must be indexed and eligible to show with a snippet to be a supporting link in AI Overviews or AI Mode. To limit what Google shows instead, use nosnippet, data-nosnippet, max-snippet or noindex.
Not for Search. Google says there are no additional requirements to appear in AI Overviews or AI Mode, and that Googlebot's robots.txt rules are the control for crawling for Search, so they rely on what Googlebot crawls. Google-Extended is a separate robots.txt token that controls Gemini training and grounding, and it has no user agent of its own.
Not reliably. Google says a page disallowed in robots.txt can still be indexed if other sites link to it, and may appear without a description. To keep a page out of Google, use a noindex meta tag or header and leave the page crawlable so Googlebot can see it.
Google says it generally caches robots.txt for up to 24 hours, longer if it can't fetch a fresh copy. To refresh it faster, use Request a recrawl in the robots.txt report in Search Console.
No. Google-Extended is a robots.txt token, not a separate crawler. Disallowing it tells Google not to use your content for training future Gemini models or for grounding in Gemini Apps and Vertex AI, and Google says it does not affect your inclusion or ranking in Google Search.
Sources
- Official documentation from Google
- Google: how to verify its crawlers
- Published IP addresses for Googlebot
- Google: AI features and your website
- Google: introduction to robots.txt
- Google: how Google interprets robots.txt
- Google: block indexing with noindex
- Google: update your robots.txt
- Cloudflare bots changelog (AI bot policy defaults)
Related AI crawlers
OAI-SearchBot · Claude-SearchBot · PerplexityBot · Google-Extended · Google-CloudVertexBot · Bingbot · Applebot · meta-webindexer
Facts checked against Google docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me