What is Googlebot?
Googlebot crawls pages for Google Search, including Discover, Images, Video and News. Google says a page must be indexed and eligible to show with a snippet to appear as a supporting link in AI Overviews or AI Mode, with no extra technical requirements.
That makes Googlebot the crawler that matters most for Google's AI answers. Google-Extended does not control AI Overviews.
- Robots.txt token:
Googlebot - Purpose: AI search & answers
- Feeds: Google Search, including AI Overviews and AI Mode
Who runs Googlebot?
Googlebot is run by Google. Everything on this page comes from Google's own documentation: Google: Google's common crawlers and Google: AI features and your website. We last checked it on 21 October 2026.
Google uses one other token, each with its own robots.txt group:
- Google-Extended: Robots.txt control token (Gemini model training and grounding)
How to allow Googlebot in robots.txt
Googlebot is allowed unless a rule blocks it. If your * group blocks it, give it its own group:
User-agent: Googlebot
Allow: /
Disallow: /admin/A named group replaces your User-agent: * group for Googlebot, so copy in any private folders you still want blocked, like /admin/ above.
How to block Googlebot in robots.txt
User-agent: Googlebot
Disallow: /Removes the blocked pages from Google Search crawling, and with it from AI Overviews and AI Mode.
To block one folder only, list just that path:
User-agent: Googlebot
Disallow: /search/Things to know about Googlebot
- Google's AI features guidance says crawling must be allowed in robots.txt and by any CDN or hosting infrastructure.
- To limit what text Google shows from a page, including in AI features, Google points to nosnippet, data-nosnippet, max-snippet and noindex, not robots.txt.
- Being eligible doesn't guarantee a page appears in AI Overviews; Google says indexing and serving aren't guaranteed.
- robots.txt only grants permission. A CDN or firewall can still block Googlebot even when your rules allow it.
Check every AI crawler at once
The checker above tests Googlebot and every other AI crawler on this list against your live robots.txt. The full AI Crawler Checker explains how to read the results. For help beyond robots.txt, see technical SEO, AI search optimisation and GEO services.
AI search & answers
- OAI-SearchBot (OpenAI): ChatGPT search answers
- Claude-SearchBot (Anthropic): Claude's search results
- PerplexityBot (Perplexity): Perplexity's search results and citations
- Applebot (Apple): Siri, Spotlight and Safari search and answers
- Bingbot (Microsoft): Bing search and Microsoft Copilot answers
User-requested fetch
- ChatGPT-User (OpenAI): pages ChatGPT opens when a user asks
- Claude-User (Anthropic): pages Claude opens when a user asks
- Perplexity-User (Perplexity): pages Perplexity opens when a user asks
Model training
- GPTBot (OpenAI): training data for OpenAI's foundation models
- ClaudeBot (Anthropic): training data for Anthropic's models
- Meta-ExternalAgent (Meta): training data for Meta's foundation AI models
- CCBot (Common Crawl): Common Crawl's open web archive, widely used as AI training data
- Amazonbot (Amazon): Amazon products and services, including Amazon AI models
Robots.txt control token
- Google-Extended (Google): Gemini model training and grounding
- Applebot-Extended (Apple): Apple foundation model training
FAQ
Googlebot questions
Only for sections that shouldn't be crawled, such as internal search results or admin areas. Blocking it site-wide removes you from Google Search and its AI features.
No. Google-Extended only controls Gemini training and grounding. Google says it doesn't affect inclusion or ranking in Google Search.
Use snippet controls such as nosnippet, data-nosnippet or max-snippet, or noindex to remove the page. A robots.txt block stops crawling, not the use of what Google already indexed.
Eligibility isn't a guarantee. The page must be indexed and snippet-eligible, and a CDN or firewall must not block Googlebot.
Done for you
Want us to fix it for you?
Send us the task. Senior marketers fix what the AI Crawler Checker found and make the rest of your site easy for Google and AI answers to understand.
