What is Claude-User?
When a Claude user asks a question that needs live web content, Claude-User retrieves the relevant page. It doesn't crawl your site on its own schedule.
Anthropic says blocking it prevents its system from retrieving your content in response to a user query, which may reduce your visibility in user-directed web search.
- Robots.txt token:
Claude-User - Purpose: User-requested fetch
- Feeds: pages Claude opens when a user asks
Who runs Claude-User?
Claude-User is run by Anthropic. Everything on this page comes from Anthropic's own documentation: Anthropic: Does Anthropic crawl data from the web?. We last checked it on 21 October 2026.
Anthropic uses 2 other tokens, each with its own robots.txt group:
- Claude-SearchBot: AI search & answers (Claude's search results)
- ClaudeBot: Model training (training data for Anthropic's models)
How to allow Claude-User in robots.txt
Claude-User is allowed unless a rule blocks it. If your * group blocks it, give it its own group:
User-agent: Claude-User
Allow: /
Disallow: /admin/A named group replaces your User-agent: * group for Claude-User, so copy in any private folders you still want blocked, like /admin/ above.
How to block Claude-User in robots.txt
User-agent: Claude-User
Disallow: /Claude can no longer open your pages when a user asks about them.
To block one folder only, list just that path:
User-agent: Claude-User
Disallow: /checkout/Things to know about Claude-User
- Anthropic says its bots, Claude-User included, honour robots.txt. Some other on-request fetchers say robots.txt may not apply to them.
- Blocking Claude-User doesn't affect ClaudeBot (training) or Claude-SearchBot (search index); each reads its own group.
- If Claude tells a user it can't open your page, check robots.txt first, then any firewall or bot-protection rule.
- robots.txt only grants permission. A CDN or firewall can still block Claude-User even when your rules allow it.
Check every AI crawler at once
The checker above tests Claude-User and every other AI crawler on this list against your live robots.txt. The full AI Crawler Checker explains how to read the results. For help beyond robots.txt, see technical SEO, AI search optimisation and GEO services.
AI search & answers
- OAI-SearchBot (OpenAI): ChatGPT search answers
- Claude-SearchBot (Anthropic): Claude's search results
- PerplexityBot (Perplexity): Perplexity's search results and citations
- Googlebot (Google): Google Search, including AI Overviews and AI Mode
- Applebot (Apple): Siri, Spotlight and Safari search and answers
- Bingbot (Microsoft): Bing search and Microsoft Copilot answers
User-requested fetch
- ChatGPT-User (OpenAI): pages ChatGPT opens when a user asks
- Perplexity-User (Perplexity): pages Perplexity opens when a user asks
Model training
- GPTBot (OpenAI): training data for OpenAI's foundation models
- ClaudeBot (Anthropic): training data for Anthropic's models
- Meta-ExternalAgent (Meta): training data for Meta's foundation AI models
- CCBot (Common Crawl): Common Crawl's open web archive, widely used as AI training data
- Amazonbot (Amazon): Amazon products and services, including Amazon AI models
Robots.txt control token
- Google-Extended (Google): Gemini model training and grounding
- Applebot-Extended (Apple): Apple foundation model training
FAQ
Claude-User questions
Yes. Anthropic says its bots, including Claude-User, honour robots.txt, even though Claude-User only visits when a person asks.
Claude can't fetch your pages to answer a user's question, which Anthropic says may reduce your visibility in user-directed web search.
Not in the usual sense. It fetches individual pages when a user's question needs them, rather than crawling your site on a schedule.
Usually a robots.txt rule that blocks Claude-User, or a CDN or firewall rule that blocks the request. This checker shows the robots.txt side.
Done for you
Want us to fix it for you?
Send us the task. Senior marketers fix what the AI Crawler Checker found and make the rest of your site easy for Google and AI answers to understand.
