---
title: "Claude-User: block or allow this AI crawler · DotsAgent"
description: "Claude-User is run by Anthropic and listed under user fetches. Its robots.txt token, whether it follows the rules, its IP list and how to control it."
url: https://dotsagent.io/agent-ready/bots/claude-user
---

Anthropic · User fetches

# What is Claude-User?

Fetches a page when a Claude user asks for it. Unlike most user-triggered fetchers, it follows robots.txt; Anthropic says all its bots do.

- **robots.txt token**: `Claude-User`
- **Operator**: Anthropic
- **Purpose**: User fetches
- **Obeys robots.txt**: The operator says it follows robots.txt.
- **Documented by operator**: Yes, on the operator's own pages.
- **Published IP list**: [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json)
- **Documentation**: [support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

## Block Claude-User from your whole site

`robots.txt`

```
User-agent: Claude-User
Disallow: /
```

## Allow it, but keep private paths out

`robots.txt`

```
User-agent: Claude-User
Allow: /
Disallow: /admin/
```

This bot fetches pages for a person, or for an agent acting on one's behalf. Several operators say robots.txt may not apply to such requests, so check the line above and enforce at your firewall if it matters.

## Other bots from Anthropic

[ClaudeBot](https://dotsagent.io/agent-ready/bots/claudebot)

[Claude-SearchBot](https://dotsagent.io/agent-ready/bots/claude-searchbot)

[Add it to a robots.txt in the generator →](https://dotsagent.io/agent-ready/robots-txt)

## Sources

1. [support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-ca

Independent reference for people who build AI agents. Not affiliated with any vendor named here.

© 2026 DotsAgent · Facts checked October 1, 2026