What is Meta-ExternalAgent?
Meta says Meta-ExternalAgent crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.
It is separate from facebookexternalhit, which fetches the title, description and image for links shared on Meta's apps.
- Robots.txt token:
meta-externalagent - Purpose: Model training
- Feeds: training data for Meta's foundation AI models
Who runs Meta-ExternalAgent?
Meta-ExternalAgent is run by Meta. Everything on this page comes from Meta's own documentation: Meta: Meta web crawlers. We last checked it on 21 October 2026.
How to allow Meta-ExternalAgent in robots.txt
Meta-ExternalAgent is allowed unless a rule blocks it. If your * group blocks it, give it its own group:
User-agent: meta-externalagent
Allow: /
Disallow: /admin/A named group replaces your User-agent: * group for Meta-ExternalAgent, so copy in any private folders you still want blocked, like /admin/ above.
How to block Meta-ExternalAgent in robots.txt
User-agent: meta-externalagent
Disallow: /Opts your pages out of Meta-ExternalAgent's crawling for AI training and product indexing. Link previews are unaffected.
To block one folder only, list just that path:
User-agent: meta-externalagent
Disallow: /downloads/Things to know about Meta-ExternalAgent
- Meta's robots.txt example uses the lowercase token meta-externalagent. Matching is case-insensitive, but the documented spelling is the safest.
- Blocking it doesn't break link previews on Meta's apps; those come from facebookexternalhit.
- Meta says robots.txt changes can take up to 24 hours because its crawlers may cache the file.
- robots.txt only grants permission. A CDN or firewall can still block Meta-ExternalAgent even when your rules allow it.
Check every AI crawler at once
The checker above tests Meta-ExternalAgent and every other AI crawler on this list against your live robots.txt. The full AI Crawler Checker explains how to read the results. For help beyond robots.txt, see technical SEO, AI search optimisation and GEO services.
AI search & answers
- OAI-SearchBot (OpenAI): ChatGPT search answers
- Claude-SearchBot (Anthropic): Claude's search results
- PerplexityBot (Perplexity): Perplexity's search results and citations
- Googlebot (Google): Google Search, including AI Overviews and AI Mode
- Applebot (Apple): Siri, Spotlight and Safari search and answers
- Bingbot (Microsoft): Bing search and Microsoft Copilot answers
User-requested fetch
- ChatGPT-User (OpenAI): pages ChatGPT opens when a user asks
- Claude-User (Anthropic): pages Claude opens when a user asks
- Perplexity-User (Perplexity): pages Perplexity opens when a user asks
Model training
Robots.txt control token
- Google-Extended (Google): Gemini model training and grounding
- Applebot-Extended (Apple): Apple foundation model training
FAQ
Meta-ExternalAgent questions
No. Link previews on Meta's apps are fetched by facebookexternalhit, a different crawler with its own robots.txt group.
Meta's docs use both, and robots.txt matching is case-insensitive. The robots.txt example in Meta's documentation uses lowercase meta-externalagent.
Meta says changes can take up to 24 hours, because its crawlers may cache your robots.txt.
Yes. Meta lists meta-webindexer for improving Meta AI search results. It has its own robots.txt group, separate from meta-externalagent.
Done for you
Want us to fix it for you?
Send us the task. Senior marketers fix what the AI Crawler Checker found and make the rest of your site easy for Google and AI answers to understand.
