---
title: "meta-webindexer: block or allow this AI crawler · DotsAgent"
description: "meta-webindexer is run by Meta and listed under search. Its robots.txt token, whether it follows the rules, its IP list and how to control it."
url: https://dotsagent.io/agent-ready/bots/meta-webindexer
---

Meta · Search

# What is meta-webindexer?

Meta's crawler for search in Meta AI. It follows robots.txt; Meta publishes no IP ranges for it.

- **robots.txt token**: `meta-webindexer`
- **Operator**: Meta
- **Purpose**: Search
- **Obeys robots.txt**: The operator says it follows robots.txt.
- **Documented by operator**: Yes, on the operator's own pages.
- **Documentation**: [developers.facebook.com](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)

## Block meta-webindexer from your whole site

`robots.txt`

```
User-agent: meta-webindexer
Disallow: /
```

## Allow it, but keep private paths out

`robots.txt`

```
User-agent: meta-webindexer
Allow: /
Disallow: /admin/
```

## Other bots from Meta

[meta-externalagent](https://dotsagent.io/agent-ready/bots/meta-externalagent)

[meta-externalfetcher](https://dotsagent.io/agent-ready/bots/meta-externalfetcher)

[meta-externalads](https://dotsagent.io/agent-ready/bots/meta-externalads)

[facebookexternalhit](https://dotsagent.io/agent-ready/bots/facebookexternalhit)

[Muse](https://dotsagent.io/agent-ready/bots/muse)

[Add it to a robots.txt in the generator →](https://dotsagent.io/agent-ready/robots-txt)

## Sources

1. [developers.facebook.com](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)/docs/sharing/webmasters/web-crawlers/

Independent reference for people who build AI agents. Not affiliated with any vendor named here.

© 2026 DotsAgent · Facts checked October 1, 2026