Meta-ExternalAgent: Meta's AI Crawler Explained

Meta-ExternalAgent is a web crawler operated by Meta. According to Meta's own crawler documentation, it crawls the web for use cases such as training foundation AI models or improving products by indexing content. It identifies itself with the user agent meta-externalagent/1.1 and can be managed with a Meta-ExternalAgent rule in robots.txt.

Quick answer: Meta-ExternalAgent is the Meta AI crawler that collects public web content for AI model training and product indexing. It is one of several documented Meta crawlers, alongside FacebookExternalHit for link previews, Meta-ExternalFetcher for user-requested fetches, and meta-webindexer for Meta AI search. You can allow or disallow it by name in robots.txt, and you can confirm real visits by checking that the IP belongs to Meta's network, AS32934, as Meta's documentation describes. Decide on each Meta crawler separately, because they serve different purposes.

What Meta-ExternalAgent Does

Meta's web crawlers documentation is the primary source. It describes Meta-ExternalAgent as a crawler used "for use cases such as training foundation AI models or improving products by indexing content." In other words, it gathers content ahead of time rather than fetching a page because a person just asked for it.

That distinction matters. A training and indexing crawler shapes what Meta's models and products know about your content over time. A user-triggered fetcher acts in the moment, on behalf of someone using a Meta product. Treating them as the same bot leads to blocking decisions with side effects you did not intend. For a wider view of how crawler types differ, see our guide to the types of AI crawlers.

• Operator: Meta
• Purpose per Meta: training foundation AI models or improving products by indexing content
• User agent: meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
• robots.txt token: Meta-ExternalAgent
• Verification: IP address within Meta's AS32934 network

How Meta-ExternalAgent Identifies Itself

Meta's documentation lists two forms of the user agent: the full string meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) and the short form meta-externalagent/1.1. Your logs may show either one, so match on the meta-externalagent token rather than the complete string.

A user agent is only a claim. Anyone can copy it, which is why user agent matching on its own is a weak test. Meta gives site owners a way to check the network side as well. Its documentation points to the list of IP prefixes announced by Meta's autonomous system, which you can pull with this command:

whois -h whois.radb.net -- '-i origin AS32934' | grep ^route

A request that carries the meta-externalagent user agent and comes from one of those prefixes is very likely genuine. A request with the right user agent from an unrelated hosting network is not. Our AI crawler verification page explains why pairing identity with network origin is the standard approach.

Meta's Crawlers Side by Side

Meta documents several crawlers, and they are easy to mix up. The table below summarizes what Meta says about each.

The practical takeaway: a rule for Meta-ExternalAgent affects AI training and product indexing only. It does not change how link previews work on Facebook or Instagram, which rely on FacebookExternalHit, and it does not cover meta-webindexer, which Meta uses for Meta AI search. If you want your pages to stay eligible for Meta AI search answers, handle meta-webindexer as its own decision. You can compare Meta's crawlers with those of other AI companies in our list of the top AI crawlers.

Managing Meta-ExternalAgent With robots.txt

Meta's documentation lets site owners manage Meta-ExternalAgent by name in robots.txt. To opt out of it across your whole site, add:

User-agent: Meta-ExternalAgent
Disallow: /

You can also disallow specific folders instead of the whole site, for example to keep premium content out while leaving public guides open. Leave rules for FacebookExternalHit and meta-webindexer untouched unless you mean to change them as well. Our guide to robots.txt for AI crawlers covers syntax, ordering, and testing.

Whatever you decide, confirm it with data. After you publish a rule, check your logs for meta-externalagent requests from Meta's network over the following days. A rule you never verified is an assumption, not a policy.

Should You Allow or Block Meta-ExternalAgent?

There is no single right answer, and the decision is yours to make on the merits of your own content.

• Allow it if you want your public content to be available to Meta's AI models and products and you are comfortable with the crawl load.
• Disallow it if you prefer that your content not be used for AI model training, while still keeping link previews and other Meta crawlers working.
• Allow it on some sections only if you want public guides available but want to keep gated or licensed content out.

Many sites make this call once and forget it. A better habit is to review crawler activity each quarter, because the pages a crawler requests tell you which content is being read.

How to Confirm Real Meta-ExternalAgent Visits

Verification comes down to two checks working together: an identity check, meaning the user agent plus a Meta-owned IP, and a record of what the crawler requested and what your server returned. Server logs hold both, but raw logs are hard to read at scale.

citAEOtion is a WordPress plugin and dashboard that shows which AI crawlers actually visit, verified from real server traffic, broken out per crawler, per page, and per status code. For Meta-ExternalAgent, that means you can see which pages Meta's crawler requested, how often, and whether it received content or errors. Agencies get multi-site dashboards and white-label reports for client sites.

If you want a clear record of how Meta's AI crawler uses your site before and after any robots.txt change, citAEOtion is a $99 Lifetime Deal, and you can book a 20-minute walkthrough to see it in action.

Frequently Asked Questions

What is Meta-ExternalAgent used for?

Meta's crawler documentation says Meta-ExternalAgent crawls the web for use cases such as training foundation AI models or improving products by indexing content. It gathers content ahead of time rather than in response to a single user request. That makes it different from Meta-ExternalFetcher, which fetches links at a user's request.

Is Meta-ExternalAgent the same as the Facebook link preview crawler?

No. Link previews for content shared on Meta's apps come from FacebookExternalHit, a separate crawler with its own user agent. A robots.txt rule for Meta-ExternalAgent does not affect link previews. Handle each Meta crawler as its own decision.

Can I block Meta-ExternalAgent with robots.txt?

Yes. Add a User-agent: Meta-ExternalAgent group with a Disallow rule for the paths you want to exclude. After publishing the rule, check your server logs for meta-externalagent requests from Meta's network to confirm the change is reflected in real traffic.

How do I know a Meta-ExternalAgent visit is genuine?

Check both the user agent and the IP address. Meta's documentation points to the IP prefixes announced by its network, AS32934, which you can list with a whois query to whois.radb.net. A request with the meta-externalagent user agent from outside those prefixes should be treated as a likely impersonator.

Does Meta-ExternalAgent run JavaScript?

Meta's crawler documentation does not say whether Meta-ExternalAgent renders JavaScript. The safe assumption for any AI crawler is that it may only read the initial HTML response. Put your key content in server-rendered HTML so every crawler can read it.