What is Claude-SearchBot?
Claude-SearchBot navigates the web to analyse content for Claude's search, so answers that use web results are more relevant and accurate.
Anthropic says blocking it prevents your content from being indexed for search optimisation, which can reduce how often your pages show up in Claude's answers.
- Robots.txt token:
Claude-SearchBot - Purpose: AI search & answers
- Feeds: Claude's search results
Who runs Claude-SearchBot?
Claude-SearchBot is run by Anthropic. Everything on this page comes from Anthropic's own documentation: Anthropic: Does Anthropic crawl data from the web?. We last checked it on 21 October 2026.
Anthropic uses 2 other tokens, each with its own robots.txt group:
- Claude-User: User-requested fetch (pages Claude opens when a user asks)
- ClaudeBot: Model training (training data for Anthropic's models)
How to allow Claude-SearchBot in robots.txt
Claude-SearchBot is allowed unless a rule blocks it. If your * group blocks it, give it its own group:
User-agent: Claude-SearchBot
Allow: /
Disallow: /admin/A named group replaces your User-agent: * group for Claude-SearchBot, so copy in any private folders you still want blocked, like /admin/ above.
How to block Claude-SearchBot in robots.txt
User-agent: Claude-SearchBot
Disallow: /Your pages are not indexed for Claude's search, which can reduce their visibility in Claude's answers.
To block one folder only, list just that path:
User-agent: Claude-SearchBot
Disallow: /internal/Things to know about Claude-SearchBot
- It is one of three Anthropic user agents, each with its own job: ClaudeBot (training), Claude-SearchBot (search) and Claude-User (on-request fetches).
- Anthropic says its bots honour robots.txt, so a Disallow rule here is respected.
- Anthropic says blocking its IP addresses may not work as an opt-out, because it stops Anthropic reading your robots.txt.
- robots.txt only grants permission. A CDN or firewall can still block Claude-SearchBot even when your rules allow it.
Check every AI crawler at once
The checker above tests Claude-SearchBot and every other AI crawler on this list against your live robots.txt. The full AI Crawler Checker explains how to read the results. For help beyond robots.txt, see technical SEO, AI search optimisation and GEO services.
AI search & answers
- OAI-SearchBot (OpenAI): ChatGPT search answers
- PerplexityBot (Perplexity): Perplexity's search results and citations
- Googlebot (Google): Google Search, including AI Overviews and AI Mode
- Applebot (Apple): Siri, Spotlight and Safari search and answers
- Bingbot (Microsoft): Bing search and Microsoft Copilot answers
User-requested fetch
- ChatGPT-User (OpenAI): pages ChatGPT opens when a user asks
- Claude-User (Anthropic): pages Claude opens when a user asks
- Perplexity-User (Perplexity): pages Perplexity opens when a user asks
Model training
- GPTBot (OpenAI): training data for OpenAI's foundation models
- ClaudeBot (Anthropic): training data for Anthropic's models
- Meta-ExternalAgent (Meta): training data for Meta's foundation AI models
- CCBot (Common Crawl): Common Crawl's open web archive, widely used as AI training data
- Amazonbot (Amazon): Amazon products and services, including Amazon AI models
Robots.txt control token
- Google-Extended (Google): Gemini model training and grounding
- Applebot-Extended (Apple): Apple foundation model training
FAQ
Claude-SearchBot questions
No. ClaudeBot collects content that may be used for model training. Claude-SearchBot indexes content for Claude's search results. Each one reads its own robots.txt group.
Anthropic says it prevents your content from being indexed for search optimisation, which may reduce your site's visibility and accuracy in Claude's search results.
Yes. Anthropic states that its bots honour robots.txt directives.
Yes. That keeps your pages available to Claude's search while opting out of training. Give each its own User-agent group.
Done for you
Want us to fix it for you?
Send us the task. Senior marketers fix what the AI Crawler Checker found and make the rest of your site easy for Google and AI answers to understand.
