---
title: "ClaudeBot: block or allow this AI crawler · DotsAgent"
description: "ClaudeBot is run by Anthropic and listed under training. Its robots.txt token, whether it follows the rules, its IP list and how to control it."
url: https://dotsagent.io/agent-ready/bots/claudebot
---

Anthropic · Training

# What is ClaudeBot?

Anthropic's crawler for collecting public pages that may be used to train Claude models. It follows robots.txt, including Crawl-delay.

- **robots.txt token**: `ClaudeBot`
- **Operator**: Anthropic
- **Purpose**: Training
- **Obeys robots.txt**: The operator says it follows robots.txt.
- **Documented by operator**: Yes, on the operator's own pages.
- **Published IP list**: [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json)
- **Documentation**: [support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

## Block ClaudeBot from your whole site

`robots.txt`

```
User-agent: ClaudeBot
Disallow: /
```

## Allow it, but keep private paths out

`robots.txt`

```
User-agent: ClaudeBot
Allow: /
Disallow: /admin/
```

## Other bots from Anthropic

[Claude-SearchBot](https://dotsagent.io/agent-ready/bots/claude-searchbot)

[Claude-User](https://dotsagent.io/agent-ready/bots/claude-user)

[Add it to a robots.txt in the generator →](https://dotsagent.io/agent-ready/robots-txt)

## Sources

1. [support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-ca

Independent reference for people who build AI agents. Not affiliated with any vendor named here.

© 2026 DotsAgent · Facts checked October 1, 2026