AI crawler and agent directory
Each AI bot we could confirm, with the token to use in robots.txt, what it fetches for, whether its operator says it follows the rules, and how to check a request really comes from it. Tokens known only from the community ai.robots.txt registry are marked as such on their own pages.
| Bot | Token | Purpose | Obeys robots.txt | Verify |
|---|---|---|---|---|
| GPTBot | GPTBot | Training | Yes | IP list |
| OAI-SearchBot | OAI-SearchBot | Search | Yes | IP list |
| ChatGPT-User | ChatGPT-User | User fetches | No | IP list |
| OAI-AdsBot | OAI-AdsBot | Ads | Not stated | IP list |
| ChatGPT agent | No token | Agents | Not stated | Signed requests |
| ClaudeBot | ClaudeBot | Training | Yes | IP list |
| Claude-SearchBot | Claude-SearchBot | Search | Yes | IP list |
| Claude-User | Claude-User | User fetches | Yes | IP list |
| Googlebot | Googlebot | Search | Yes | IP list |
| Google-Extended | Google-Extended | Training | Yes | — |
| Google-CloudVertexBot | Google-CloudVertexBot | Search | Yes | IP list |
| GoogleOther | GoogleOther | Training | Yes | IP list |
| Google-Agent | Google-Agent | Agents | No | IP list |
| Google-GeminiNotebook | Google-GeminiNotebook | User fetches | No | IP list |
| Gemini Spark | No token | Agents | Not stated | — |
| Gemini-Deep-Research | Gemini-Deep-Research | User fetches | Not stated | — |
| GoogleAgent-Mariner | GoogleAgent-Mariner | Agents | Not stated | — |
| Applebot | Applebot | Search | Yes | IP list |
| Applebot-Extended | Applebot-Extended | Training | Yes | — |
| PerplexityBot | PerplexityBot | Search | Yes | IP list |
| Perplexity-User | Perplexity-User | User fetches | No | IP list |
| meta-externalagent | meta-externalagent | Training | Yes | — |
| meta-webindexer | meta-webindexer | Search | Yes | — |
| meta-externalfetcher | meta-externalfetcher | User fetches | No | — |
| meta-externalads | meta-externalads | Ads | Yes | — |
| facebookexternalhit | facebookexternalhit | User fetches | No | — |
| Muse | No token | Agents | Not stated | — |
| Amazonbot | Amazonbot | Training | Yes | IP list |
| Amzn-SearchBot | Amzn-SearchBot | Search | Yes | IP list |
| Amzn-User | Amzn-User | User fetches | No | IP list |
| DuckAssistBot | DuckAssistBot | Search | Yes | IP list |
| MistralAI-User | MistralAI-User | User fetches | Yes | IP list |
| MistralAI-Index | MistralAI-Index | Search | Yes | IP list |
| MistralAI-Training | MistralAI-Training | Training | Yes | — |
| CCBot | CCBot | Training | Yes | IP list |
| cohere-ai | cohere-ai | User fetches | Not stated | — |
| Bytespider | Bytespider | Training | No | — |
| YouBot | YouBot | Search | Not stated | — |
| Diffbot | Diffbot | Search | Not stated | — |
| Timpibot | Timpibot | Training | Not stated | — |
What the purposes mean
- Training
- Collects pages for model training or general research crawling. Google-Extended and Applebot-Extended are control tokens that never fetch anything themselves.
- Search
- Indexes pages for a search product, from Google Search to AI answer engines such as ChatGPT search, Perplexity and DuckAssist. Blocking one keeps you out of that product.
- User fetches
- Fetches a page because a person asked for it in a chat or tool. Several operators say robots.txt may not apply to these requests.
- Agents
- Browses and acts on a user's behalf, often in a full browser. Many have no token and can be identified only by signature, if at all.
- Ads
- Checks ad landing pages that advertisers submitted. These are left out of the robots.txt generator.
Check a bot is who it says it is
A user-agent string costs nothing to fake. Most large operators publish the IP ranges their bots use as JSON files, marked IP list in the table. Compare the request's address with the list for that token, and treat a match on the name alone as unverified.
Some operators also document reverse DNS. Common Crawl, for example, says CCBot hosts resolve to *.crawl.commoncrawl.org; confirm that the hostname resolves back to the same address before you trust it.
Agents increasingly sign requests with HTTP Message Signatures (RFC 9421) under the Web Bot Auth draft. The request names its operator in Signature-Agent, and you verify it with the public keys at that domain's /.well-known/http-message-signatures-directory. A valid signature proves the operator, not the user behind it or what that user may do.
Build a robots.txt from this list →
Questions about AI crawlers
Which AI crawlers ignore robots.txt?
By their operators' own documentation, ChatGPT-User, Perplexity-User, Google-Agent, Google-GeminiNotebook, Amzn-User, meta-externalfetcher and facebookexternalhit may not follow it, mostly because a user triggered the fetch. Bytespider is reported by the community not to respect it.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot collects pages for training OpenAI models, OAI-SearchBot indexes pages for ChatGPT search, and ChatGPT-User fetches a page when a person asks for it. The first two follow robots.txt; OpenAI says its rules may not apply to the third.
Which AI agents have no robots.txt token?
ChatGPT agent, Gemini Spark and Meta's Muse. ChatGPT agent can be verified through its Web Bot Auth signature from https://chatgpt.com. Gemini Spark in local Chrome mode looks like the user's own browser, and Muse has no published user agent.
How do I verify a request really comes from Googlebot?
Google publishes Googlebot's IP ranges as a JSON file, linked from its entry in this directory. Check the request's address against those ranges rather than trusting the user-agent string.
Where does this list of AI bots come from?
Each entry links to the operator's documentation where it exists, and tokens without it come from the community ai.robots.txt registry. We left out DeepSeekBot, Kimi-User, Manus-User, NovaAct, Devin and PetalBot because we found no operator documentation for them. The list was checked on 1 October 2026.