What is ClaudeBot?
ClaudeBot collects public web content that could contribute to training Anthropic's generative AI models. Blocking it signals that your future material should be excluded from training datasets.
It doesn't power Claude's search or on-request page fetches; those are Claude-SearchBot and Claude-User.
- Robots.txt token:
ClaudeBot - Purpose: Model training
- Feeds: training data for Anthropic's models
Who runs ClaudeBot?
ClaudeBot is run by Anthropic. Everything on this page comes from Anthropic's own documentation: Anthropic: Does Anthropic crawl data from the web?. We last checked it on 21 October 2026.
Anthropic uses 2 other tokens, each with its own robots.txt group:
- Claude-SearchBot: AI search & answers (Claude's search results)
- Claude-User: User-requested fetch (pages Claude opens when a user asks)
How to allow ClaudeBot in robots.txt
ClaudeBot is allowed unless a rule blocks it. If your * group blocks it, give it its own group:
User-agent: ClaudeBot
Allow: /
Disallow: /admin/A named group replaces your User-agent: * group for ClaudeBot, so copy in any private folders you still want blocked, like /admin/ above.
How to block ClaudeBot in robots.txt
User-agent: ClaudeBot
Disallow: /Tells Anthropic to exclude your future content from model training. Claude's search and on-request fetches are unaffected.
To block one folder only, list just that path:
User-agent: ClaudeBot
Disallow: /library/Things to know about ClaudeBot
- Anthropic supports the non-standard Crawl-delay directive for ClaudeBot, so you can slow it down instead of blocking it.
- Anthropic says blocking its IP addresses may not work, because that stops it reading your robots.txt; use robots.txt rules.
- Blocking ClaudeBot while allowing Claude-SearchBot keeps you in Claude's search without contributing to training.
- robots.txt only grants permission. A CDN or firewall can still block ClaudeBot even when your rules allow it.
Check every AI crawler at once
The checker above tests ClaudeBot and every other AI crawler on this list against your live robots.txt. The full AI Crawler Checker explains how to read the results. For help beyond robots.txt, see technical SEO, AI search optimisation and GEO services.
AI search & answers
- OAI-SearchBot (OpenAI): ChatGPT search answers
- Claude-SearchBot (Anthropic): Claude's search results
- PerplexityBot (Perplexity): Perplexity's search results and citations
- Googlebot (Google): Google Search, including AI Overviews and AI Mode
- Applebot (Apple): Siri, Spotlight and Safari search and answers
- Bingbot (Microsoft): Bing search and Microsoft Copilot answers
User-requested fetch
- ChatGPT-User (OpenAI): pages ChatGPT opens when a user asks
- Claude-User (Anthropic): pages Claude opens when a user asks
- Perplexity-User (Perplexity): pages Perplexity opens when a user asks
Model training
- GPTBot (OpenAI): training data for OpenAI's foundation models
- Meta-ExternalAgent (Meta): training data for Meta's foundation AI models
- CCBot (Common Crawl): Common Crawl's open web archive, widely used as AI training data
- Amazonbot (Amazon): Amazon products and services, including Amazon AI models
Robots.txt control token
- Google-Extended (Google): Gemini model training and grounding
- Applebot-Extended (Apple): Apple foundation model training
FAQ
ClaudeBot questions
Yes. Anthropic supports Crawl-delay for ClaudeBot. For example, Crawl-delay: 1 in the ClaudeBot group limits how often it requests pages.
No. Citations in search come from Claude-SearchBot and on-request visits from Claude-User. ClaudeBot only covers training data.
Anthropic says blocking its IP addresses may not work correctly, because it stops Anthropic reading your robots.txt. Use robots.txt rules instead.
Yes. Add Disallow lines for those folders in a ClaudeBot group and leave everything else allowed.
Done for you
Want us to fix it for you?
Send us the task. Senior marketers fix what the AI Crawler Checker found and make the rest of your site easy for Google and AI answers to understand.
