How the facebookexternalhit Crawler Works

Q: How do I force Meta to re-crawl a URL?

Run it through the Sharing Debugger at /tools/debug/ , which Meta names as the way to trigger a scrape, or call the Sharing API. Meta's own best practices page describes the Debugger as the tool for sites that need to update their tags more often than the standard 24 hour update period.

What is facebookexternalhit?

Meta sends facebookexternalhit when a link to your site is shared on one of its apps, and the crawler fetches the page so Meta can build the preview card. Meta's Meta Web Crawlers documentation puts it plainly: the primary purpose "is to crawl the content of an app or website that was shared on one of Meta's family of apps, such as Facebook, Instagram, or Messenger." The same page says the crawler "gathers, caches, and displays information about the app or website such as its title, description, and thumbnail image."

The share does not have to come from a share button. Meta says the link "might have been shared by copying and pasting or by using the Facebook social plugin," so a URL typed into a Messenger thread triggers the same fetch as a public post.

What user agent strings does facebookexternalhit send?

Meta documents two, and both start with the same lowercase token. The page says the UA string in your log files "will be similar to one of the following":

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
facebookexternalhit/1.1

Note the mismatch between the prose and the wire format. Meta's headings and body text spell the crawler FacebookExternalHit in camel case, while the string your server actually receives is lowercase. Match on a case-insensitive substring, not on the camel case spelling.

Meta also publishes the exact request to replay when you want to reproduce what the crawler sees:

curl -v --compressed -H "Range: bytes=0-524288" -H "Connection: close" \
  -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" "$URL"

How is facebookexternalhit different from meta-externalagent?

They are separate user agents with separate purposes, and a robots.txt rule against one does not touch the other. Meta now documents five crawlers on the same page, which is the split that matters if you want AI training traffic gone without losing your link previews.

Crawler	User agent token	Purpose Meta states
FacebookExternalHit	`facebookexternalhit/1.1`	Crawls content shared on Meta's family of apps and builds the preview
Meta-WebIndexer	`meta-webindexer/1.1`	Navigates the web to improve Meta AI search result quality
Meta-ExternalAds	`meta-externalads/1.1`	Crawls for advertising and other business-related products and services
Meta-ExternalAgent	`meta-externalagent/1.1`	Training foundation AI models or improving products by indexing content directly
Meta-ExternalFetcher	`meta-externalfetcher/1.1`	Fetches individual links at a user's request, supporting agentic AI

Blocking meta-externalagent costs you nothing in the feed, because the preview card is built by a different crawler. Meta's own robots.txt example uses meta-externalagent as the sample agent:

User-agent: meta-externalagent
Allow: /                    # Allow everything
Disallow: /private/         # Disallow a specific directory

Meta publishes no date for when the AI crawlers were separated out of facebookexternalhit. The page carries a single "Updated: May 21, 2026" line and no revision history, so the split is documented as a fact of the present with no before and after to check.

A server rack with network cables, representing the backend infrastructure that must respond to crawler requests within seconds.

What does Meta require from your server?

Five requirements, all listed under Crawler Requirements, and four of them are easy to fail without noticing. Your server "must use gzip and deflate encodings." Any Open Graph properties "need to be listed before the first 1 MB of your website or app, or it will be cutoff." The content has to be crawlable "within a few seconds or Facebook will be unable to display the content."

The fourth is the one that trips up range-aware servers. Meta says your app or website "should either generate and return a response with all required properties according to the bytes specified in the Range header of the crawler request or it should ignore the Range header altogether." A server that honours Range: bytes=0-524288 but puts the <head> after that boundary returns a technically correct response with nothing to parse. The fifth is the allow-list, covered below.

None of this is about what your tags say, only whether the crawler can reach them. The tag contents are a separate problem, and the full set of properties Meta and every other network reads is covered in this walkthrough of Open Graph tags.

How do you tell a real facebookexternalhit request from a fake one?

By IP, because the user agent string is a free-text header anyone can send. Meta agrees and says to allow-list "either the user agent strings or the IP addresses (more secure) used by the crawler."

Then the documentation stops. That phrase links to an anchor named identify, and the live page has no IP address section at all. In the page source the same identify anchor is attached to four different headings, including the robots.txt section. Meta tells you the more secure option and then points at nothing.

The command is published, on an unrelated page. Meta's WhatsApp webhooks documentation gives the query for Meta's own routes:

AdaptlyPost

–

Schedule your content across all platforms

Start 7-Day FREE Trial

All-platform analytics

Social Inbox

AI-powered assistant

whois -h whois.radb.net -- '-i origin AS32934' | grep '^route' | awk '{print $2}' | sort

That autonomous system number checks out. ARIN's registry lists AS32934 as ASName: FACEBOOK, registered 24 August 2004. Run the query yourself and you get the current set of route objects: on 12 September 2026 it returned 416 unique IPv4 prefixes and 633 unique IPv6 prefixes, from 102.132.100.0/24 through blocks like 2401:db00::/32. Meta's own caveat travels with the command, since the WhatsApp page warns that "Meta periodically changes its IP addresses." A reverse DNS check on the requesting address is the cheaper alternative for most sites.

Confirming a real facebookexternalhit request

Match the user agent string. A free-text header, anyone can send it, no proof on its own.

Allow-list the IP. The option Meta calls more secure, which then links to an anchor with no IP list on it.

Run a reverse DNS check. The cheaper alternative for most sites.

Meta names all three checks but leaves the IP list undocumented.

Can you block facebookexternalhit in robots.txt?

You can, and Meta says the block will not always hold. The documentation states that "the FacebookExternalHit crawler might bypass robots.txt when performing security or integrity checks, such as checking for malware or malicious content." Meta-ExternalFetcher carries the same carve-out because it fetches links a person asked for.

Meta also asks for patience on the rule itself: "Please allow up to 24 hours for changes to robots.txt to take effect because crawlers may cache the contents of robots.txt for up to 24 hours." A disallow you added this morning does not explain a crawl this afternoon.

Blocking the crawler kills the preview. The link still posts, it just arrives as a bare URL with no card, which is the same visual outcome as a broken tag. The click that follows still goes through Meta's redirect layer either way, which is a separate hop with its own effects on your analytics and is covered in this breakdown of the l.facebook.com link shim.

Why does the crawler return code 0 or a blank preview?

Because of a DNS record most sites never think about. Meta's Link Sharing FAQ states it flatly: "The crawler will look for a AAAA record and return a response code 0 if it is not found." An IPv4-only host that works in every browser can return nothing to facebookexternalhit.

Three other failure modes come straight out of the same FAQ. Images are "cached asynchronously, so the image may not render the first time someone shares your content," and Meta's fix is to pre-cache by triggering a scrape with the Sharing Debugger at /tools/debug/. An image that renders as a white box is "no longer available, is too big or could not be fetched," and the ceiling is 8 MB. An image between 200 x 200 and 600 x 315 pixels renders as a small square rather than a card.

The last one is permanent. Once a link has been shared and "there have been more than 50 interactions with the post (comments, likes, shares, etc.), then the title cannot be changed." Meta's stated reason is to stop a site swapping the story out from under people who already engaged with it. Every other property stays editable. Get the title right before the link goes out, because after 50 interactions you are editing a field nobody will see updated.

When the same URL goes to several networks at once, adaptlypost publishes one post to multiple accounts, and each network then runs its own crawler against that one URL on its own schedule. The crawl behaviour stays the platform's. It is the same division of labour you see with link wrapping, where X rewrites every URL into a t.co link the moment you post it, and where your own campaign tracking has to survive the rewrite, which is the argument for putting UTM parameters on the URL before it ever reaches a composer.

Frequently asked questions

Does facebookexternalhit execute JavaScript?

Meta does not say. The documentation describes a fetch with a Range header, a few seconds of tolerance and a 1 MB cutoff on where Open Graph properties can sit, and never mentions rendering. Serve your og: tags in the server response rather than injecting them client side.

Is facebookexternalhit the same crawler Instagram and Messenger use?

Yes. Meta names all three in one sentence, saying the crawler handles content "shared on one of Meta's family of apps, such as Facebook, Instagram, or Messenger." One user agent covers the preview crawl across those surfaces.

A magnifying glass held over lines of code on a screen, evoking the log inspection needed to catch traffic bursts.

Why is facebookexternalhit hitting my server in bursts?

Meta publishes no crawl rate, no request budget and no crawl-delay support for it. The documentation lists purpose, user agent strings, server requirements and robots.txt handling, and stops there. Rate limiting by IP against the AS32934 route set is the only lever the documentation leaves you.

Should I allow facebookexternalhit but block meta-externalagent?

That combination is what the separate user agents exist for. Allowing facebookexternalhit keeps preview cards working on Facebook, Instagram and Messenger, while a Disallow: / under User-agent: meta-externalagent addresses the crawler Meta describes as training foundation AI models.

What does the Range header on a crawler request mean?

It asks your server for a byte window instead of the whole document, and Meta's sample request uses Range: bytes=0-524288, the first 512 KB. Meta accepts either behaviour: honour the header and return the requested bytes with all required properties inside them, or ignore the header entirely and send the full response.

How do I force Meta to re-crawl a URL?

Run it through the Sharing Debugger at /tools/debug/, which Meta names as the way to trigger a scrape, or call the Sharing API. Meta's own best practices page describes the Debugger as the tool for sites that need to update their tags more often than the standard 24 hour update period.

AdaptlyPost

–

Schedule your content across all platforms

Start 7-Day FREE Trial

All-platform analytics

Social Inbox

AI-powered assistant

Does a link shared privately in a Messenger chat get crawled the same way as a public post?

Meta says a share can come from copying and pasting a link or from the Facebook social plugin, not just a share button. A URL pasted into a Messenger thread triggers the same facebookexternalhit fetch as a link posted publicly on Facebook. Meta documents no separate crawl path for a private message, only the family of apps a share can come from.

Why does facebookexternalhit appear lowercase in my server logs?

Meta's documentation spells the crawler FacebookExternalHit in headings and body text, but the string it actually sends is lowercase: facebookexternalhit/1.1. A filter or allow-list rule built around the camel case spelling misses real requests. Match on a case-insensitive substring instead.

What encoding does facebookexternalhit require from my server?

Meta's crawler requirements state that your server must use gzip and deflate encodings. That sits alongside the rest of the list: Open Graph tags inside the first 1 MB, a page crawlable within a few seconds, correct handling of the Range header, and the IP allow-list option. Meta lists gzip and deflate first and states the requirement without qualification.

Does Meta ever ignore my robots.txt disallow rule for facebookexternalhit?

Meta says facebookexternalhit might bypass robots.txt when it runs security or integrity checks, such as scanning for malware or malicious content. Meta-ExternalFetcher carries the same carve-out, since it fetches links a person asked for directly. A disallow rule also is not instant: Meta asks for up to 24 hours because crawlers can cache robots.txt for that long.