AI Crawler Checker: is GPTBot blocked on your site?

How the checker reads your robots.txt

We fetch /robots.txt from your site (https first, then http) and apply the same rules crawlers use, as written in the robots.txt standard, RFC 9309:

• Group selection. A crawler follows every group that names its token, case-insensitively. Only if none names it does it fall back to the User-agent: * group. We show this as “Not mentioned (inherits *)”.
• Longest match wins. Among the Allow and Disallow rules that match your path, the longest pattern decides. If an Allow and a Disallow are the same length, Allow wins.
• Wildcards. * matches any characters and a trailing $ means “ends here”, so Disallow: /*.pdf$ blocks PDFs only.
• Missing or broken files. A 404 means everything is allowed. A server error or timeout means crawlers treat the whole site as “don't crawl”.

Every row shows the deciding rule and its line number, so you can find it in your file straight away.

How to read your results

The headline counts the crawlers that decide whether you can be found and cited in AI answers.

• AI search & answers: crawlers that index pages for AI search, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot. Googlebot and Bingbot are here too, because AI Overviews and Copilot are built on the Google and Bing indexes.
• User-requested fetch: agents that open a page when someone asks an assistant about it. OpenAI and Perplexity say robots.txt may not apply to theirs, and the row says so.
• Model training: crawlers that collect training data, such as GPTBot, ClaudeBot and CCBot. These are shown in neutral colours, because blocking them is a business choice, not an error.
• Robots.txt control tokens: Google-Extended and Applebot-Extended never visit your site. They tell Google and Apple whether content their search crawlers collect may be used for AI training.

Should you block AI training crawlers?

It depends on what you publish. Many businesses block training crawlers and allow AI search crawlers: they stay findable and citable in ChatGPT, Claude and Perplexity answers without adding their content to future training data. Others allow everything, because being part of what models learn can help them recognise the brand. Both are valid; the snippet builder writes either policy for you.

What is rarely intended is blocking AI search crawlers. That removes you from the answers your buyers already read, while competitors stay in them.

Common fixes

1. Delete a site-wide Disallow: / under User-agent: * that was meant for a staging site.
2. Give AI search crawlers their own group with Allow: /, and copy in the private-folder rules from your * group, because a named group replaces it.
3. Make /robots.txt return 200 or 404, never a server error, and serve it as plain text rather than an HTML page.
4. Check your CDN or firewall's bot settings: robots.txt only grants permission, it can't get a crawler past a block.

If you'd rather not touch these files yourself, our technical SEO team fixes crawl and robots.txt issues, and AI search optimisation covers what comes after: making the pages worth citing.

AI crawler guides

What each crawler does, who runs it and how to allow or block it, based on each operator's own documentation. List last reviewed 21 October 2026.