# Googlebot

> Google's main search crawler. Google says Googlebot's robots.txt rules are the control for crawling for Search, and its AI features (AI Overviews and AI Mode) only link to pages that are indexed and eligible to show with a snippet.

Canonical page: https://getrankroot.com/ai-bots/googlebot · Facts checked against Google docs on 2026-10-20

| Operator | Used for | Type | Obeys robots.txt |
| --- | --- | --- | --- |
| Google | Google Search (incl. AI Overviews) | AI search index | Yes |

## Should you allow it?

Always allow it.

**If you block it:** Google can't crawl your pages for Search, Discover, Images, Video or News, and they can't be shown with content in AI Overviews or AI Mode. A blocked URL can still be listed without a description if other sites link to it.

The first string is Googlebot Smartphone, then Googlebot Desktop; Google says the last two are used rarely. W.X.Y.Z is the Chrome version, which changes over time, so match it with a wildcard.

## robots.txt rules

Allow Googlebot:

```
User-agent: Googlebot
Allow: /
```

Block Googlebot:

```
User-agent: Googlebot
Disallow: /
```

Disallow Googlebot from one folder only:

```
User-agent: Googlebot
Disallow: /private/
Allow: /
```

A bot follows the most specific User-agent group that names it and ignores the generic `User-agent: *` group, so repeat any shared Disallow lines inside it.

## Same company, different bots: Google

| Bot | Type | Purpose | Obeys robots.txt | If you block it |
| --- | --- | --- | --- | --- |
| [Google-Extended](https://getrankroot.com/ai-bots/google-extended.md) | Control token (does not crawl) | Gemini training and grounding | Token only | Google won't use your content to train future Gemini models or for grounding in Gemini Apps and Vertex AI. Google Search is unaffected. |
| [Googlebot](https://getrankroot.com/ai-bots/googlebot.md) | AI search index | Google Search (incl. AI Overviews) | Yes | Google can't crawl your pages for Search, Discover, Images, Video or News, and they can't be shown with content in AI Overviews or AI Mode. A blocked URL can still be listed without a description if other sites link to it. |
| [Google-CloudVertexBot](https://getrankroot.com/ai-bots/google-cloudvertexbot.md) | User-triggered fetcher | Vertex AI Agents | Yes | Vertex AI agents that site owners request can't crawl your site. No effect on Google Search. |

## User agent string

```
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html)
```

## How to verify it's really Googlebot

Find its requests in your server logs:

```
grep -i "Googlebot" access.log
```

Google's common crawlers such as Googlebot resolve to hostnames like crawl-***.googlebot.com or geo-crawl-***.geo.googlebot.com.

Reverse DNS check (Google documents this): the hostname of the IP must end in `googlebot.com` or `google.com` or `googleusercontent.com`, and a forward lookup of that hostname must return the same IP.

Published IP list: https://developers.google.com/static/crawling/ipranges/common-crawlers.json

## Common mistakes

- Using Disallow to hide a page from Google. Google says robots.txt is not a way to keep a page out of Google: a disallowed URL can still be indexed if other sites link to it. Use noindex instead.
- Combining noindex with a robots.txt Disallow. Google says the page must not be blocked by robots.txt, or Googlebot never sees the noindex rule.
- Setting Crawl-delay for Googlebot. Google supports only user-agent, allow, disallow and sitemap; crawl-delay is ignored.
- Writing a User-agent: Googlebot group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the Googlebot group.
- Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
- Trusting the user agent header alone. Anyone can send "Googlebot" in a request; verify the source IP as shown above before allowlisting or blocking.

## FAQ

### What is Googlebot?

Google's main search crawler. Google says Googlebot's robots.txt rules are the control for crawling for Search, and its AI features (AI Overviews and AI Mode) only link to pages that are indexed and eligible to show with a snippet.

### Does Googlebot respect robots.txt?

Yes, according to Google.

### What happens if I block Googlebot?

Google can't crawl your pages for Search, Discover, Images, Video or News, and they can't be shown with content in AI Overviews or AI Mode. A blocked URL can still be listed without a description if other sites link to it.

### Should I block Googlebot?

Always allow it.

### Does blocking Googlebot remove my site from AI Overviews and AI Mode?

Effectively yes, along with the rest of Google Search. Google says Googlebot's robots.txt rules are the control for crawling for Search, and a page must be indexed and eligible to show with a snippet to be a supporting link in AI Overviews or AI Mode. To limit what Google shows instead, use nosnippet, data-nosnippet, max-snippet or noindex.

### Is there a separate Googlebot for AI?

Not for Search. Google says there are no additional requirements to appear in AI Overviews or AI Mode, and that Googlebot's robots.txt rules are the control for crawling for Search, so they rely on what Googlebot crawls. Google-Extended is a separate robots.txt token that controls Gemini training and grounding, and it has no user agent of its own.

### Will Disallow: / for Googlebot remove my pages from Google?

Not reliably. Google says a page disallowed in robots.txt can still be indexed if other sites link to it, and may appear without a description. To keep a page out of Google, use a noindex meta tag or header and leave the page crawlable so Googlebot can see it.

### How fast does Google pick up robots.txt changes?

Google says it generally caches robots.txt for up to 24 hours, longer if it can't fetch a fresh copy. To refresh it faster, use Request a recrawl in the robots.txt report in Search Console.

### Is Google-Extended the same as Googlebot?

No. Google-Extended is a robots.txt token, not a separate crawler. Disallowing it tells Google not to use your content for training future Gemini models or for grounding in Gemini Apps and Vertex AI, and Google says it does not affect your inclusion or ranking in Google Search.

## Sources

- [Official documentation from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [Google: how to verify its crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
- [Published IP addresses for Googlebot](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)
- [Google: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)
- [Google: introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro)
- [Google: how Google interprets robots.txt](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)
- [Google: block indexing with noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing)
- [Google: update your robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/submit-updated-robots-txt)

Check whether Googlebot can reach a site: https://getrankroot.com/tools/ai-crawler-checker or `GET https://getrankroot.com/api/bots?url=example.com`. Full list: https://getrankroot.com/ai-bots.md