What is Google-Extended?
Google-Extended lets you manage whether content Google crawls from your site may be used to train future Gemini models, and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.
Google says it has no separate user agent string. Crawling happens with Google's existing crawlers, and the token is only read from robots.txt.
- Robots.txt token:
Google-Extended - Purpose: Robots.txt control token
- Feeds: Gemini model training and grounding
Who runs Google-Extended?
Google-Extended is run by Google. Everything on this page comes from Google's own documentation: Google: Google's common crawlers. We last checked it on 21 October 2026.
Google uses one other token, each with its own robots.txt group:
- Googlebot: AI search & answers (Google Search, including AI Overviews and AI Mode)
How to allow Google-Extended in robots.txt
Google-Extended is allowed unless a rule blocks it, which means Google may use your content for Gemini model training and grounding. To state it explicitly:
User-agent: Google-Extended
Allow: /
Disallow: /admin/A named group replaces your User-agent: * group for Google-Extended, so copy in any private folders you still want blocked, like /admin/ above.
How to block Google-Extended in robots.txt
User-agent: Google-Extended
Disallow: /Opts your content out of Gemini training and grounding. Google Search, AI Overviews and rankings are unaffected.
To block one folder only, list just that path:
User-agent: Google-Extended
Disallow: /research/Things to know about Google-Extended
- You won't see Google-Extended in your server logs, because it never visits your site.
- Google says Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal.
- AI Overviews and AI Mode use the Google Search index, so they are controlled through Googlebot and snippet settings, not Google-Extended.
- robots.txt only grants permission. A CDN or firewall can still block Google-Extended even when your rules allow it.
Check every AI crawler at once
The checker above tests Google-Extended and every other AI crawler on this list against your live robots.txt. The full AI Crawler Checker explains how to read the results. For help beyond robots.txt, see technical SEO, AI search optimisation and GEO services.
AI search & answers
- OAI-SearchBot (OpenAI): ChatGPT search answers
- Claude-SearchBot (Anthropic): Claude's search results
- PerplexityBot (Perplexity): Perplexity's search results and citations
- Googlebot (Google): Google Search, including AI Overviews and AI Mode
- Applebot (Apple): Siri, Spotlight and Safari search and answers
- Bingbot (Microsoft): Bing search and Microsoft Copilot answers
User-requested fetch
- ChatGPT-User (OpenAI): pages ChatGPT opens when a user asks
- Claude-User (Anthropic): pages Claude opens when a user asks
- Perplexity-User (Perplexity): pages Perplexity opens when a user asks
Model training
- GPTBot (OpenAI): training data for OpenAI's foundation models
- ClaudeBot (Anthropic): training data for Anthropic's models
- Meta-ExternalAgent (Meta): training data for Meta's foundation AI models
- CCBot (Common Crawl): Common Crawl's open web archive, widely used as AI training data
- Amazonbot (Amazon): Amazon products and services, including Amazon AI models
Robots.txt control token
- Applebot-Extended (Apple): Apple foundation model training
FAQ
Google-Extended questions
No. AI Overviews use pages indexed in Google Search, and Google says Google-Extended doesn't affect inclusion in Search.
It has no separate user agent string. Google crawls with its usual crawlers and reads Google-Extended from robots.txt as a control token.
Whether Google may use your content to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.
No. Google says it isn't used as a ranking signal in Google Search.
Done for you
Want us to fix it for you?
Send us the task. Senior marketers fix what the AI Crawler Checker found and make the rest of your site easy for Google and AI answers to understand.
