AI crawler list · Google · User-triggered fetcher

Google-CloudVertexBot

Crawls sites when a site owner asks Google Cloud to build a Vertex AI agent from them. Google says it has no effect on Google Search or other products.

Facts checked against Google docs on Oct 20, 2026.

Operator

Google

Used for

Vertex AI Agents

Type

User-triggered fetcher

Obeys robots.txt

Yes

Is Google-CloudVertexBot allowed on your site?

Reads your robots.txt and requests your page as each AI bot. Free, no signup.

Should you allow it?

Allow it if you or your team build Vertex AI agents on your own content; otherwise it rarely matters.

If you block it: Vertex AI agents that site owners request can't crawl your site. No effect on Google Search.

robots.txt rules

Allow Google-CloudVertexBot

User-agent: Google-CloudVertexBot
Allow: /

Block Google-CloudVertexBot

User-agent: Google-CloudVertexBot
Disallow: /

Disallow Google-CloudVertexBot from one folder only

User-agent: Google-CloudVertexBot
Disallow: /private/
Allow: /

Put each User-agent group in the robots.txt at your domain root. A bot follows the most specific group that names it and ignores the generic User-agent: * group, so repeat any shared Disallow lines inside it. Build a full file with the AI robots.txt generator.

Same company, different bots: Google

Google uses separate user agents for separate jobs. Blocking one does not block the others.

Bot	Purpose	Obeys robots.txt	If you block it
`Google-Extended` Control token (does not crawl)	Gemini training and grounding	Token only	Google won't use your content to train future Gemini models or for grounding in Gemini Apps and Vertex AI. Google Search is unaffected.
`Googlebot` AI search index	Google Search (incl. AI Overviews)	Yes	Google can't crawl your pages for Search, Discover, Images, Video or News, and they can't be shown with content in AI Overviews or AI Mode. A blocked URL can still be listed without a description if other sites link to it.
`Google-CloudVertexBot` User-triggered fetcher	Vertex AI Agents	Yes	Vertex AI agents that site owners request can't crawl your site. No effect on Google Search.

Allow Google search and user fetches, opt out of Google training

User-agent: Googlebot
User-agent: Google-CloudVertexBot
Allow: /

User-agent: Google-Extended
Disallow: /

How to verify it's really Google-CloudVertexBot

Find its requests in your server logs (the path to your access log depends on your host):

grep -i "Google-CloudVertexBot" access.log

Google's documentation does not publish the full user agent string for Google-CloudVertexBot, so treat a token match as a starting point and confirm the IP.

Google's common crawlers such as Googlebot resolve to hostnames like crawl-***.googlebot.com or geo-crawl-***.geo.googlebot.com.

Reverse DNS check (Google documents this): look up the IP from your log, confirm the hostname ends in googlebot.com, google.com or googleusercontent.com, then look up that hostname and confirm it returns the same IP.

host 203.0.113.7          # replace with the IP from your log
host <hostname-from-above>  # must return the same IP

Common mistakes

Writing a User-agent: Google-CloudVertexBot group and expecting your User-agent: * rules to still apply. A crawler follows only the most specific group that matches it and ignores the others, so repeat any shared Disallow lines inside the Google-CloudVertexBot group.
Leaving a CDN or host "block AI bots" setting on and only checking robots.txt. Those settings block requests at the network edge. Cloudflare, for example, changed its defaults on September 15, 2026 for customers who did not opt out: bots it classifies as Training or Agent are blocked on pages that display ads, and multi-purpose crawlers that combine Search and Training are affected by the Training block.
Trusting the user agent header alone. Anyone can send "Google-CloudVertexBot" in a request; verify the source IP as shown above before allowlisting or blocking.

FAQ

What is Google-CloudVertexBot?

Crawls sites when a site owner asks Google Cloud to build a Vertex AI agent from them. Google says it has no effect on Google Search or other products.

Does Google-CloudVertexBot respect robots.txt?

Yes, according to Google.

What happens if I block Google-CloudVertexBot?

Vertex AI agents that site owners request can't crawl your site. No effect on Google Search.

Should I block Google-CloudVertexBot?

Allow it if you or your team build Vertex AI agents on your own content; otherwise it rarely matters.

Sources

Related AI crawlers

ChatGPT-User · OAI-AdsBot · Claude-User · Perplexity-User · Google-Extended · Googlebot · meta-externalfetcher · Amzn-User

Facts checked against Google docs on Oct 20, 2026. Spot an error? rankroot.support@proton.me