Google-Extended Explained: What Blocking It Really Does

Google-Extended is a robots.txt product token that controls whether content Google crawls from your website may train future Gemini models and ground answers in Gemini Apps and Vertex AI. Blocking it does not remove a site from AI Overviews, because Google says it does not affect inclusion in Google Search, and AI Overviews are part of Search.

• Google-Extended has 0 HTTP user agent strings of its own: it is a control token, and the crawling is done by Google's existing crawlers (Google's list of common crawlers).
• The token governs 2 uses: training future Gemini models, and grounding answers in Gemini Apps and Grounding with Google Search on Vertex AI.
• Google generally caches a robots.txt file for up to 24 hours and ignores anything after the first 500 KiB (Google's robots.txt specification).
• The Search generative AI control, open to all websites since 31 August 2026, is the setting for AI Overviews and AI Mode, and Google says it does not affect AI training (Search Console Help on the Search generative AI control).

What does Google-Extended control?

Google-Extended controls two things: whether your content may train future generations of Gemini models, and whether it may be used for grounding in Gemini Apps and on Vertex AI.

Training is the familiar part. Content that Google crawls can be used to teach future Gemini models, which then power Gemini Apps and the Vertex AI API for Gemini, and a Disallow rule for Google-Extended asks Google not to use your pages for that. The models learn from text long before anyone asks them a question, so this choice shapes what future models know about your subject rather than what any engine cites today.

Grounding is the part many owners overlook. Google defines it as supplying content from the Google Search index to the model at the moment a question is asked, in Gemini Apps and in Grounding with Google Search on Vertex AI. Blocking Google-Extended therefore asks Google to keep your pages out of those live, grounded answers as well, which is a real cost for a business that wants to be mentioned when someone asks the Gemini app a question.

Is Google-Extended a separate crawler?

No. Google-Extended has no user agent string of its own; Google crawls with its existing user agents, and the token exists only so that robots.txt can express a choice.

That design has practical effects. You will never see a visit from "Google-Extended" in your server logs, because the pages are fetched by crawlers such as Googlebot, and the token only tells Google what it may do with what was fetched. A firewall rule or CDN setting therefore cannot single out Google-Extended, and any attempt to block it by user agent or IP address would hit Googlebot and remove you from Search instead.

The robots.txt line is the only control Google documents for this token. Google's crawler list gives an example in which Google-Extended is allowed one archive folder and disallowed the rest of the archive, which shows that the token accepts ordinary Allow and Disallow rules for paths. Write it as its own group, with User-agent: Google-Extended on the first line and your rules beneath.

Does blocking Google-Extended remove a site from AI Overviews?

No. Google states that Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal, and AI Overviews and AI Mode are features of Search.

Google's list of crawlers does not name AI Overviews in its Google-Extended entry, so the answer rests on how Google describes the features themselves. Its page on AI features says that robots.txt rules for Googlebot are the control site owners use to manage crawl access for Search, AI features included, and it sends owners to Google-Extended only for training and grounding in some of Google's other systems. Google AI Overviews optimization is therefore a matter of Googlebot, indexing and snippets, and the Google-Extended line plays no part in it.

The reverse also holds. A website that blocks Google-Extended stays eligible for AI Overviews and AI Mode as long as Googlebot can crawl it, its pages are indexed and snippet-eligible, and its Search Console property is set to include generative AI features. Owners who block the token expecting to vanish from AI Overviews will see no change there, and owners who feared losing those links by blocking it can relax.

How do you add Google-Extended to robots.txt?

Add a separate group that names Google-Extended as the user agent, followed by Disallow rules for the paths it should not use; leave Googlebot's rules untouched.

Google's robots.txt specification says each crawler follows only one group, the one with the most specific user agent that matches it, and that groups naming the same user agent are combined. A group for Google-Extended therefore changes nothing for Googlebot, and the reverse is true too. The table shows the common choices.

Goal	Lines to add to robots.txt	Effect on AI Overviews
Opt the whole website out of Gemini training and grounding	User-agent: Google-Extended then Disallow: /	None
Opt out one folder, such as paid guides	User-agent: Google-Extended then Disallow: /guides/	None
Allow one folder inside a blocked one	User-agent: Google-Extended then Allow: /archive/public/ and Disallow: /archive/	None
Allow training and grounding	No Google-Extended group at all	None

Do not rely on a User-agent: * group to cover the token. Google's specification says specific groups and the * group are not combined, so naming Google-Extended explicitly is the only way to be sure of the rule that applies. Google generally caches robots.txt for up to 24 hours, so allow at least a day before expecting a change to be read.

Should a business block Google-Extended?

For most businesses that want customers to find them, allowing Google-Extended is the simpler choice; blocking suits publishers whose writing is the product they sell.

The case for blocking is about control of your work. A publisher, a training provider or a writer may not want their text used to train models that could later answer readers without sending them anywhere. Google provides the token for exactly that choice, and using it costs nothing in Google Search.

The case for allowing is about being known. A hypothetical bakery in Leeds benefits when future models already know its name and specialities, and when Gemini Apps can ground an answer in its current pages. Blocking the token gives up both, while leaving AI Overviews unchanged, so a business should block it only when keeping its content out of model training matters more than appearing in Gemini's answers.

Which control removes a website from AI Overviews?

The Search generative AI control in Search Console removes a website's content and links from AI Overviews and AI Mode; nosnippet and noindex work page by page.

Search Console Help describes the control under Settings, with Include as the default for every property. Setting it to Exclude keeps your links and content out of AI Overviews, AI Mode and generative AI features in Discover, usually within 1 to 2 days, and Google says it is not a ranking or inclusion signal for the rest of Search. The same page says the setting does not affect AI training and points to Google-Extended for that, which shows how cleanly Google separates the two choices.

For single pages, the robots meta rules apply. The nosnippet rule stops a page's content being used as a direct input for AI Overviews and AI Mode, at the cost of its snippet in ordinary results, and noindex removes the page from Search entirely. A business that wants out of both training and AI Overviews needs two settings: the Google-Extended line in robots.txt and the Exclude choice in Search Console.

How can you check what your robots.txt tells Google?

Open your robots.txt file at the root of your domain and read every group, then confirm Googlebot is allowed and that any Google-Extended group says what you intended.

Look for three common mistakes. A Disallow: / meant for Google-Extended can end up under Googlebot or under * after an edit, a security plugin can add its own rules, and an old staging file can survive a launch. Each of these blocks far more than training, and the first two can remove the whole website from Search.

To check every engine at once, scan your website with the AI Search Visibility Checker. It reads robots.txt for each engine's search bot, asks for the homepage as each bot to catch firewalls, and lists training bots such as Google-Extended apart from the search bots that decide citations. The method page explains how each part of the score is worked out.

Questions owners ask about Google-Extended

Will blocking Google-Extended lower my Google rankings?

No. Google states that Google-Extended is not used as a ranking signal in Google Search and does not affect a site's inclusion there.

Does Google-Extended affect Google Discover or Google Images?

Google's crawler list ties Google-Extended only to Gemini training and grounding, while Googlebot and its image and video crawlers serve Search, Images and Discover. Generative AI features in Discover are covered by the Search generative AI control instead.

Can I block Google-Extended for only part of my website?

Yes, the token accepts ordinary path rules. Name it in its own group and add Disallow lines for the folders to keep out, with Allow lines for any exceptions inside them.

Is Google-Extended the same as GPTBot or ClaudeBot?

It plays a similar role, since all three express a training choice, but it is not a crawler. GPTBot and ClaudeBot are bots that visit pages, while Google-Extended is a token read by Google's existing crawlers.

What to decide about Google-Extended this week

Decide whether model training and Gemini grounding help or hurt your business, write that choice into robots.txt, and leave Googlebot's access untouched.

If you sell your writing, add a Google-Extended group that disallows the folders that hold it, and keep your marketing pages open if you still want Gemini to mention you. If you want to be found, leave the token out and spend the time on the pages Google's AI features quote. Either way, confirm the Search generative AI control is set the way you want, because that setting, not Google-Extended, decides whether your pages appear in AI Overviews.