The short answer
Google-Extended is a robots.txt control token, not a separate crawler. It lets you tell Google not to use your content for training Gemini models or for grounding Gemini answers. It does not affect Google Search, your rankings or Google AI Overviews. Those rely on Googlebot and normal Search controls.
- Google-Extended is a permission setting in robots.txt, not a bot that visits your site.
- It controls use of your content for Gemini model training and grounding.
- It has no effect on Google Search rankings or AI Overviews.
- To limit AI Overviews, you use Search controls such as nosnippet, which also affect normal results.
How does Google-Extended work?
Googlebot crawls your pages for Google Search, as it always has. Google-Extended does not crawl anything itself.
Instead, it is a name you can use in robots.txt. A rule for Google-Extended tells Google whether content it already crawls may be used for Gemini.
Because it is only a token, you will not see Google-Extended in your server logs.
Google introduced it in 2023, as publishers asked for more say over AI use of their content. Google's crawler documentation lists it alongside its other user agents.
What does Google-Extended control?
The token covers Gemini model training and grounding. Grounding means Gemini looking up web content to support an answer.
| Google feature | Main control | Affected by Google-Extended? |
|---|---|---|
| Google Search rankings | Googlebot and robots.txt | No |
| Google AI Overviews | Googlebot and Search controls like nosnippet | No |
| Gemini model training | Google-Extended | Yes |
| Grounding in Gemini answers | Google-Extended | Yes |
Google updates these rules from time to time, so check its crawler documentation before changing anything.
How to use Google-Extended in robots.txt
To opt out, add a group for Google-Extended. Keep Googlebot allowed, or you will drop out of Google Search too.
You can also opt out for part of a site. Use a folder path, such as Disallow: /research/, instead of the whole site.
# Opt out of Gemini training and grounding User-agent: Google-Extended Disallow: / # Keep Google Search working as normal User-agent: Googlebot Allow: /
Should you block Google-Extended?
It depends on how much you value Gemini visibility. Because the token covers grounding, blocking it may reduce how often Gemini can draw on your pages.
Gemini is a separate product from Google Search, with its own app and audience. Losing grounding there could mean fewer chances to be cited when people ask Gemini directly.
Publishers with licensing concerns sometimes opt out. Most businesses chasing AI visibility leave it allowed.
WebValet Pro tracks Gemini answers to your buyer prompts each week. That helps you see whether a change in this setting coincides with a change in visibility.
How do you control AI Overviews instead?
AI Overviews are part of Google Search. Google says they use the same controls as normal search snippets.
The nosnippet robots meta tag, the data-nosnippet attribute and max-snippet limit what Google can show. These affect your regular search snippets too, so use them with care.
Google also says no special markup is needed to appear in AI Overviews. Normal SEO best practice applies.
At the time of writing, Google offers no setting that removes you from AI Overviews alone. Most businesses that want visibility leave these controls off their key pages.
Common Google-Extended mistakes
These misunderstandings come up often.
- Searching server logs for Google-Extended visits, which never appear.
- Expecting it to remove your site from AI Overviews.
- Blocking Googlebot to opt out of AI, which removes you from Search.
- Assuming it controls Microsoft Copilot, which draws on Bing instead.
Google-Extended vs other AI control tokens
Google is not the only company to separate search crawling from AI use. Apple offers a similar token called Applebot-Extended.
Applebot-Extended lets you opt out of Apple using your content to train its AI models. Like Google-Extended, it does not stop Applebot from crawling for search features.
OpenAI, Anthropic and Perplexity take a different approach. They run separate named crawlers for training, search and user requests, which you allow or block one by one.
Whichever company you deal with, read its current documentation before you edit rules. Names and scopes have changed before and may change again.
Questions people ask
Is Google-Extended a crawler?
No. It is a control token used in robots.txt. Google's existing crawlers do the fetching, and the token only sets how the content may be used.
Will blocking Google-Extended hurt my Google rankings?
No. Google says it does not affect Search rankings or inclusion. Search is controlled through Googlebot, which the token does not touch.
Does Google-Extended remove me from AI Overviews?
No. AI Overviews are part of Search and follow Search controls. Use nosnippet or related settings, knowing they also affect normal snippets.
Does blocking Google-Extended remove content already used?
It applies going forward. It does not undo training that has already happened.
How do I check whether my site blocks Google-Extended?
Open yourdomain.com/robots.txt and look for a Google-Extended group. If there is none, Gemini use is allowed by default. WebValet's free check also reads your robots.txt rules for the main AI bots and tokens.