What is ClaudeBot?

Anthropic's crawler for collecting public pages that may be used to train Claude models. It follows robots.txt, including Crawl-delay.

robots.txt token

ClaudeBot

Operator

Anthropic

Purpose

Training

Obeys robots.txt

●Yes The operator says it follows robots.txt.

Documented by operator

Yes, on the operator's own pages.

Published IP list

claude.com/crawling/bots.json

Documentation

support.claude.com

Block ClaudeBot from your whole site

User-agent: ClaudeBot
Disallow: /

Allow it, but keep private paths out

User-agent: ClaudeBot
Allow: /
Disallow: /admin/

Other bots from Anthropic

Add it to a robots.txt in the generator →