What is CCBot?
CCBot is run by Common Crawl, a non-profit that maintains a free, open repository of web crawl data anyone can use. Researchers and companies, including AI developers, use the archive.
Blocking CCBot keeps your pages out of future crawls. It can't remove pages already published in past archives.
- Robots.txt token:
CCBot - Purpose: Model training
- Feeds: Common Crawl's open web archive, widely used as AI training data
Who runs CCBot?
CCBot is run by Common Crawl. Everything on this page comes from Common Crawl's own documentation: Common Crawl: CCBot. We last checked it on 21 October 2026.
How to allow CCBot in robots.txt
CCBot is allowed unless a rule blocks it. If your * group blocks it, give it its own group:
User-agent: CCBot
Allow: /
Disallow: /admin/A named group replaces your User-agent: * group for CCBot, so copy in any private folders you still want blocked, like /admin/ above.
How to block CCBot in robots.txt
User-agent: CCBot
Disallow: /Your pages are left out of future Common Crawl archives. Past archives are unaffected.
To block one folder only, list just that path:
User-agent: CCBot
Disallow: /customers/Things to know about CCBot
- Some crawlers falsely identify themselves as CCBot. Common Crawl says real CCBot runs on dedicated IP ranges and resolves under crawl.commoncrawl.org.
- Common Crawl publishes CCBot's IP ranges as JSON, so you can verify or allowlist it.
- For legal opt-out requests, Common Crawl keeps an Opt-Out Ledger in addition to robots.txt.
- robots.txt only grants permission. A CDN or firewall can still block CCBot even when your rules allow it.
Check every AI crawler at once
The checker above tests CCBot and every other AI crawler on this list against your live robots.txt. The full AI Crawler Checker explains how to read the results. For help beyond robots.txt, see technical SEO, AI search optimisation and GEO services.
AI search & answers
- OAI-SearchBot (OpenAI): ChatGPT search answers
- Claude-SearchBot (Anthropic): Claude's search results
- PerplexityBot (Perplexity): Perplexity's search results and citations
- Googlebot (Google): Google Search, including AI Overviews and AI Mode
- Applebot (Apple): Siri, Spotlight and Safari search and answers
- Bingbot (Microsoft): Bing search and Microsoft Copilot answers
User-requested fetch
- ChatGPT-User (OpenAI): pages ChatGPT opens when a user asks
- Claude-User (Anthropic): pages Claude opens when a user asks
- Perplexity-User (Perplexity): pages Perplexity opens when a user asks
Model training
- GPTBot (OpenAI): training data for OpenAI's foundation models
- ClaudeBot (Anthropic): training data for Anthropic's models
- Meta-ExternalAgent (Meta): training data for Meta's foundation AI models
- Amazonbot (Amazon): Amazon products and services, including Amazon AI models
Robots.txt control token
- Google-Extended (Google): Gemini model training and grounding
- Applebot-Extended (Apple): Apple foundation model training
FAQ
CCBot questions
Anyone. The archive is free and open, and it's used by researchers and companies, including those training AI models.
No. robots.txt only affects future crawls. Common Crawl handles legal opt-out requests through its Opt-Out Ledger.
Use a reverse DNS lookup: real CCBot hosts resolve under crawl.commoncrawl.org. Common Crawl also publishes its IP ranges as JSON.
No. CCBot isn't a search engine crawler. Google and Bing use Googlebot and Bingbot, which have their own rules.
Done for you
Want us to fix it for you?
Send us the task. Senior marketers fix what the AI Crawler Checker found and make the rest of your site easy for Google and AI answers to understand.
