Google-Extended is a robots.txt control that lets site owners decide whether content Google crawls from their site can be used to train future Gemini models and to ground answers in some of Google's AI products. It isn't a separate crawler. It's a token you name in robots.txt to tell Google how it may use what its existing crawlers already fetch.
For businesses, Google-Extended is the main way to opt out of Google's AI training without hurting search visibility. It's also easy to misunderstand. It doesn't control AI Overviews or AI Mode, and blocking it has trade-offs for visibility in the Gemini app.
What is Google-Extended?
Google introduced Google-Extended on September 28, 2023, as a control publishers could use to manage whether their sites help improve Bard and Vertex AI generative APIs. Bard has since become Gemini, and the documentation has been updated to match.
Google's current list of common crawlers describes Google-Extended as a standalone product token. It manages whether content Google crawls can be used to train the Gemini models that power Gemini Apps and the Vertex AI API for Gemini, and whether it can be used for grounding in those products.
Two details matter most. First, "Google-Extended doesn't have a separate HTTP request user agent string." Google crawls with its existing user agents, and the token is used in a control capacity. Second, Google states that "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Google added that ranking language in an April 2025 documentation update, as Search Engine Journal reported.
Why Google-Extended matters
It separates AI training from search. Without a separate token, the only way to keep content away from Google entirely would be to block Googlebot, which would also remove you from Search. Google-Extended lets you make a separate choice for Gemini training.
It also covers grounding. Google describes grounding as "providing content from the Google Search index to the model at prompt time to improve factuality and relevancy." Blocking Google-Extended means your content shouldn't be used that way in Gemini Apps or Grounding with Google Search on Vertex AI. For a brand that wants to show up in Gemini app answers, that's a real trade-off.
It doesn't cover Search's AI features. Google's guidance on AI features and your website says that to limit the information shown from your pages in Search, you use nosnippet, data-nosnippet, max-snippet or noindex. Google-Extended is mentioned separately, as the way "to limit AI training and grounding in some of Google's other systems."
How Google-Extended works
A control token, not a crawler
Googlebot and Google's other crawlers fetch your pages as usual. When it comes to how that content may be used for Gemini, Google checks your robots.txt rules for the Google-Extended token. Because there's no separate user agent, you won't see Google-Extended in your server logs.
Writing the rule
You add Google-Extended to robots.txt like any other user agent. To opt your whole site out, add a group with User-agent: Google-Extended and Disallow: /. Google's own example shows you can also be selective, disallowing a folder while allowing a specific path inside it.
What it doesn't change
Blocking Google-Extended doesn't change how Googlebot crawls or indexes your site, and Google says it isn't a ranking signal. It also applies going forward: a new rule doesn't remove content from models that were already trained.
How to decide on Google-Extended
- List what you care about. Search traffic, citations in AI Overviews, visibility in the Gemini app and control over training can pull in different directions.
- Remember the grounding trade-off. If being cited in Gemini app answers matters to you, think carefully before blocking.
- Use path rules for sensitive sections. You can allow marketing pages while disallowing premium content or paid research.
- Manage AI Overviews separately. If you want to limit how your content appears in Search's AI features, look at snippet controls, and weigh that against the visibility you'd lose.
- Keep your policy consistent across AI companies. Other companies use their own tokens, such as OpenAI's GPTBot. Decide on a policy and apply it to each one deliberately.
Common mistakes and misconceptions
"Blocking Google-Extended keeps me out of AI Overviews." It doesn't. AI Overviews and AI Mode are part of Google Search, and Google points to Search controls like snippet rules and noindex for those.
"Blocking Google-Extended hurts my rankings." Google says it doesn't affect inclusion in Search and isn't used as a ranking signal.
"I should look for Google-Extended in my logs." There's no separate Google-Extended user agent, so it won't appear in access logs.
"Blocking Googlebot is a safer way to opt out of AI." Blocking Googlebot stops Google Search from crawling your pages, which costs far more visibility than it protects.
Google-Extended vs. Googlebot
Googlebot is the crawler that fetches pages for Google Search. Google-Extended is a permission setting that governs whether that crawled content can also be used for Gemini training and grounding. Blocking Googlebot affects Search. Blocking Google-Extended doesn't.
How Butter approaches Google-Extended
At Butter, we look at Gemini through both doors: Google Search and the Gemini app. We'll walk you through robots.txt choices like Google-Extended so you don't block visibility by accident. In our GEO plan, monthly AI prompt testing and indexing strategy include reviewing which AI user agents your robots.txt and hosting setup allow.
We don't make the training opt-out decision for you. We explain the trade-offs and let you choose. On the GEO+SEO plan, robots.txt problems fall under the technical fixes we handle. Our GEO service page has the full details.
Frequently asked questions
Does Google-Extended affect my Google Search rankings?
No. Google states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. It only controls whether crawled content can be used for Gemini training and grounding.
Does blocking Google-Extended remove my site from AI Overviews?
No. AI Overviews and AI Mode are part of Google Search. To limit how your content appears there, Google points to snippet and indexing controls such as nosnippet, max-snippet and noindex.
Is Google-Extended a separate crawler?
No. Google-Extended has no separate user agent string. Google's existing crawlers fetch your pages, and the Google-Extended token in robots.txt tells Google how that content may be used.
How do I block Google-Extended?
Add a group to your robots.txt file with User-agent: Google-Extended followed by Disallow: / to opt out your whole site. You can also disallow specific folders instead of the entire site.