Google-Extended

What it controls, and what it does not

Google-Extended has no HTTP user-agent string of its own. Crawling still happens through Google's existing crawlers, Googlebot among them; the robots.txt token only acts as a usage control on content Google has already crawled. Google's own documentation describes it as a token web publishers can use "to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding" in Gemini Apps and in Grounding with Google Search on Vertex AI.

The same documentation states plainly: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Blocking Google-Extended changes nothing about whether a page shows up in Google Search results, in AI Overviews, or in AI Mode. That coverage runs through Googlebot and the ordinary Search indexing rules instead; it is a separate decision from the Gemini training and grounding uses the token actually governs.

How to use it

A publisher who wants to keep content out of the Gemini uses the token covers writes a dedicated User-agent block in robots.txt:

User-agent: Google-Extended
Disallow: /

Because it is a standalone token rather than a general-purpose crawler name, it does not inherit from or get overridden by a User-agent: Googlebot rule, and a User-agent: * wildcard rule applies to it the same way it applies to any other named token with no more specific rule of its own.

Why it matters for AI visibility

The distinction Google draws, training and grounding versus Search inclusion, is easy to get backwards, and the consequence of getting it backwards runs in only one direction: a publisher who disallows Google-Extended expecting it to also affect visibility in Google Search or AI Overviews will see no change there at all, while a publisher who wants their content excluded specifically from Gemini's training data or from what Gemini can ground an answer on has exactly one lever for that, this token.

This has become more relevant as AI-visibility work increasingly asks which systems a brand's content can feed into, and under what terms. Google-Extended answers that question for one specific pair of uses, Gemini training and Gemini grounding, and says nothing about the much larger question of whether the same content shows up in Google's own AI Overviews, which draws on the regular Search index instead.

Good practices

• Treat the decision to allow or disallow Google-Extended as separate from any decision about Googlebot or Search visibility; they do not affect each other.
• Check Google's current documentation before acting, since the scope of what Google-Extended covers has changed before and the exact products it governs can shift as Google ships new Gemini features.
• Write the rule as its own User-agent block rather than assuming a general AI-crawler block covers it, since it is a distinct token.
• Decide at the level of what you actually want: keeping content out of Gemini's training data is a different goal from keeping it out of Gemini's grounding, even though one robots.txt rule currently covers both.

Common mistakes

• Disallowing Google-Extended and assuming it reduces a page's visibility in Google Search, AI Overviews, or AI Mode; Google states explicitly that it does not.
• Assuming Google-Extended has its own crawler and user-agent string; it does not, Google's existing crawlers do the fetching, and the token only governs usage afterward.
• Confusing Google-Extended with a general "block all AI training" switch; it names one company's specific products, not a universal setting.
• Not revisiting the rule after Google updates the scope of what Google-Extended covers, since the list of affected products is not fixed.