What is OAI-SearchBot?
OAI-SearchBot crawls the web so ChatGPT's search features can surface and link to websites. If your robots.txt blocks it, OpenAI says your pages won't be shown in ChatGPT search answers, although they can still appear as plain navigational links.
It is separate from GPTBot. OAI-SearchBot is about being found in ChatGPT; GPTBot is about model training. You can allow one and block the other.
- Robots.txt token:
OAI-SearchBot - Purpose: AI search & answers
- Feeds: ChatGPT search answers
Who runs OAI-SearchBot?
OAI-SearchBot is run by OpenAI. Everything on this page comes from OpenAI's own documentation: OpenAI: Overview of OpenAI crawlers. We last checked it on 21 October 2026.
OpenAI uses 2 other tokens, each with its own robots.txt group:
- ChatGPT-User: User-requested fetch (pages ChatGPT opens when a user asks)
- GPTBot: Model training (training data for OpenAI's foundation models)
How to allow OAI-SearchBot in robots.txt
OAI-SearchBot is allowed unless a rule blocks it. If your * group blocks it, give it its own group:
User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/A named group replaces your User-agent: * group for OAI-SearchBot, so copy in any private folders you still want blocked, like /admin/ above.
How to block OAI-SearchBot in robots.txt
User-agent: OAI-SearchBot
Disallow: /Your pages stop being shown in ChatGPT search answers (they may still appear as navigational links).
To block one folder only, list just that path:
User-agent: OAI-SearchBot
Disallow: /account/Things to know about OAI-SearchBot
- OpenAI says search changes can take about 24 hours to take effect after you update robots.txt.
- OpenAI recommends allowing OAI-SearchBot and its published IP ranges, so a firewall or bot-protection rule doesn't block it even when robots.txt allows it.
- If you allow both OAI-SearchBot and GPTBot, OpenAI may use the results of one crawl for both purposes to avoid crawling twice.
- robots.txt only grants permission. A CDN or firewall can still block OAI-SearchBot even when your rules allow it.
Check every AI crawler at once
The checker above tests OAI-SearchBot and every other AI crawler on this list against your live robots.txt. The full AI Crawler Checker explains how to read the results. For help beyond robots.txt, see technical SEO, AI search optimisation and GEO services.
AI search & answers
- Claude-SearchBot (Anthropic): Claude's search results
- PerplexityBot (Perplexity): Perplexity's search results and citations
- Googlebot (Google): Google Search, including AI Overviews and AI Mode
- Applebot (Apple): Siri, Spotlight and Safari search and answers
- Bingbot (Microsoft): Bing search and Microsoft Copilot answers
User-requested fetch
- ChatGPT-User (OpenAI): pages ChatGPT opens when a user asks
- Claude-User (Anthropic): pages Claude opens when a user asks
- Perplexity-User (Perplexity): pages Perplexity opens when a user asks
Model training
- GPTBot (OpenAI): training data for OpenAI's foundation models
- ClaudeBot (Anthropic): training data for Anthropic's models
- Meta-ExternalAgent (Meta): training data for Meta's foundation AI models
- CCBot (Common Crawl): Common Crawl's open web archive, widely used as AI training data
- Amazonbot (Amazon): Amazon products and services, including Amazon AI models
Robots.txt control token
- Google-Extended (Google): Gemini model training and grounding
- Applebot-Extended (Apple): Apple foundation model training
FAQ
OAI-SearchBot questions
Yes. OpenAI treats each user agent separately, so you can block GPTBot to opt out of training and still allow OAI-SearchBot so ChatGPT search can show and cite your pages.
OpenAI says it takes about 24 hours for its search systems to adjust after you change robots.txt.
Only as a navigational link. OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, but can still appear as links when someone looks for them by name.
robots.txt only grants permission. A CDN, firewall or bot-protection setting can still block the request. Allowlist OpenAI's published IP ranges for OAI-SearchBot in those tools.
Done for you
Want us to fix it for you?
Send us the task. Senior marketers fix what the AI Crawler Checker found and make the rest of your site easy for Google and AI answers to understand.
