Aditya Sharma

Google-Extended User Agent: It Is a Token, Not a Crawler

On this page, 8 sections

Google-Extended is not a user agent in the sense most people mean. It is a robots.txt product token with no HTTP user agent string of its own, so it will never show up in your access logs. Google says so directly: crawling is done with existing Google user agent strings, and the token is used in a control capacity.

What the token controls is whether content Google has already crawled can be used to train future generations of Gemini models, and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google also says it does not affect inclusion in Google Search and is not a ranking signal.

I read Google’s crawler documentation on 11 October 2026. The page footer showed it was last updated on 14 July 2026. Everything below is from that page unless I say otherwise.

What Google-Extended actually is

Google’s crawler overview splits its clients into three groups: common crawlers, special-case crawlers and user-triggered fetchers. Google-Extended sits on the common crawlers list, but it never makes a request of its own.

Most entries on that list have a “User-Agent in HTTP requests” block with a string you can grep for. Googlebot has several. GoogleOther has two. Google-Extended, like Googlebot-News, has none: its block says it has no separate string.

The consequence is simple. When you write User-agent: Google-Extended in robots.txt, you are not naming a visitor. You are flipping a switch Google reads for content its existing crawlers have already fetched.

Google’s own example shows the shape of the rule:

user-agent: Google-Extended
allow: /archive/1Q84
disallow: /archive/

What it controls, and what it does not

The documentation lists the affected products in one sentence. Google-Extended is a standalone product token that publishers use to manage whether content Google crawls from their sites may be used for:

  • Training future generations of Gemini models that power Gemini Apps and the Vertex AI API for Gemini.
  • Grounding, which Google defines as providing content from the Google Search index to the model at prompt time, in Gemini Apps and Grounding with Google Search on Vertex AI.

Then the sentence that matters for SEO: Google-Extended does not impact a site’s inclusion in Google Search, and it is not used as a ranking signal in Google Search.

That is the whole claim. The page does not say anything about other Google surfaces beyond that list, so I am not going to guess about them. If a post tells you that Google-Extended controls a surface that is not named in the section above, ask where it read that.

Why your logs show nothing

Because there is no string, there is nothing to count. Searching your logs for “Google-Extended”, finding zero hits and concluding Google is not crawling you for Gemini is a wrong reading in both directions.

The requests that feed the product come from Googlebot and its siblings. Google documents that the common crawlers generally come from the IP ranges in its common-crawlers.json file, and that the reverse DNS name of the host matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com.

So the check that works is the one for Googlebot, not for the token. If you want to know whether a hit is really Google, resolve the IP and look at the hostname.

What to write in robots.txt

If you want to keep your content out of Gemini training and grounding, one group is enough:

User-agent: Google-Extended
Disallow: /

If you want to allow it explicitly, which some sites do so a policy is visible in the file, the group is:

User-agent: Google-Extended
Allow: /

Google says neither line affects your inclusion in Search. Googlebot keeps following the rules written for Googlebot, or for * if there is no Googlebot group.

The group-matching trap

The easy mistake is not about Google-Extended itself. It is about what happens to the * group once a specific group exists.

Under RFC 9309, a crawler that finds a group naming it obeys that group only. The * group is never consulted. If you list Google-Extended in a shared group with twenty other AI agents and give that group a single Allow: /, you have just discarded every Disallow you wrote under * for that token.

My own robots.txt lists Google-Extended in a shared group of AI user agents. I wrote up the consequence in robots.txt for AI crawlers: the user agents that matter, where I found my own file permitted GPTBot to crawl a path that Googlebot could not. Read that before you paste a long list of tokens into one group.

The other Google tokens you will meet

Google-Extended is the one people ask about, but the same page lists several neighbours that matter when you read a log or write a rule.

  • GoogleOther: the generic crawler that product teams may use for fetching publicly accessible content, for example one-off research and development crawls. Crawling preferences for it do not affect any specific product. It has real user agent strings, and variants for images and video (GoogleOther-Image, GoogleOther-Video).
  • Google-CloudVertexBot: affects crawls requested by site owners for building Vertex AI Agents. Google says it has no effect on Google Search or other products.
  • Googlebot: the Search crawler. Blocking it is how you leave Search, which is a different decision from anything above.

The practical rule: a token that appears in robots.txt but not in logs (Google-Extended) is a policy switch. A token that appears in both (GoogleOther, Google-CloudVertexBot, Googlebot) is also a visitor you can measure. For how to count them, see reading WordPress access logs.

The other AI vendors work differently. OpenAI and Anthropic name separate bots, one for training and others for search or user requests. I cover those in GPTBot vs OAI-SearchBot vs ChatGPT-User and ClaudeBot, Claude-User and Claude-SearchBot.

Questions people ask

Is Google-Extended a crawler?

No. Google’s documentation says it has no separate HTTP user agent string and that crawling is done with existing Google user agent strings. It is a robots.txt token only.

Does blocking Google-Extended hurt my SEO?

Google states that it does not impact a site’s inclusion in Google Search and is not used as a ranking signal. That is Google’s statement. I have not run a controlled test, and I would not claim one exists.

How do I verify Google-Extended in my logs?

You cannot, because it never makes a request. Verify Googlebot instead, using the IP ranges in common-crawlers.json or a reverse DNS lookup that ends in googlebot.com.

Does it stop Googlebot from crawling my site?

Google says it does not affect your inclusion in Search. It does not say the token changes what Googlebot crawls. What it controls is how Google may use content its existing crawlers have fetched.

How to confirm a Google visitor is real

Google adds a caution on the same page: the HTTP user agent string can be spoofed. A log line that says Googlebot proves nothing by itself.

The documented check is a reverse DNS lookup on the requesting IP. The hostname should match crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. Then resolve that hostname forward and confirm it returns the same IP.

host <ip-from-your-log>
host <hostname-returned-above>

This matters for Google-Extended in one specific way. Since the token never appears as a visitor, nobody can fake it, and nobody can be blocked by it either. The only things you can spoof or verify are the Google clients that actually send requests.

Do this today

Open your robots.txt and search for Google-Extended. Check which group it sits in and whether that group’s rules are the ones you meant. If it shares a group with other tokens, confirm that the group does not accidentally override a Disallow you wrote under *.

Resources

Tell me where I am wrong

Your email is not published and I do not add it to any list. Corrections with a source are the ones I act on fastest.

More in AI

Every piece in AI

  1. AI Memory With Tagging: Tags, Embeddings and Graph Edges Are Not Interchangeable MemPalace ships five backends and the default one has a dash in the Namespaces column. Which tools expose tags, and where a tag beats… AI 10 min
  2. AI Memory for Writers: What Survives the Round Trip This category was built for code. One project targets prose, it has 66 stars, and the test that eliminates the rest takes two minutes. AI 9 min
  3. codebase-memory-mcp vs Graphify: What Each One Actually Stores Both are alive, both refuse a vector database, and both were pushed to this week. The split is who owns the graph file. AI 10 min
  4. Choosing an AI Memory Layer: Sort Them by What Comes Back, Not by Stars A 24,587-star repo whose README says the code moved elsewhere. One axis that actually separates these tools, and the field sorted on it. AI 9 min
  5. AI Memory Platform Comparison: Eleven Projects, Six Columns, Every Cell Sourced LightRAG's own README says its default storage is not for production. Storage, persistence, retrieval, licence and release date for eleven projects. AI 9 min
  6. Structured Data and AI Answers: What Schema Is Actually Doing for You Now Google says schema is not required for generative AI search. What it does document, what practitioners assume, and how to audit your own graph. AI 8 min