---
title: "Diffbot: block or allow this AI crawler · DotsAgent"
description: "Diffbot is run by Diffbot and listed under search. Its robots.txt token, whether it follows the rules, its IP list and how to control it."
url: https://dotsagent.io/agent-ready/bots/diffbot
---

Diffbot · Search

# What is Diffbot?

Diffbot's crawler for data extraction and its knowledge graph. The token comes from community lists, not Diffbot's own docs.

- **robots.txt token**: `Diffbot`
- **Operator**: Diffbot
- **Purpose**: Search
- **Obeys robots.txt**: The operator does not say.
- **Documented by operator**: No. The token is known only from community lists such as ai.robots.txt.

Community sources say whether it honours robots.txt depends on Diffbot's own users.

## Block Diffbot from your whole site

`robots.txt`

```
User-agent: Diffbot
Disallow: /
```

## Allow it, but keep private paths out

`robots.txt`

```
User-agent: Diffbot
Allow: /
Disallow: /admin/
```

[Add it to a robots.txt in the generator →](https://dotsagent.io/agent-ready/robots-txt)

## Sources

1. [github.com](https://github.com/ai-robots-txt/ai.robots.txt)/ai-robots-txt/ai.robots.txt

Independent reference for people who build AI agents. Not affiliated with any vendor named here.

© 2026 DotsAgent · Facts checked October 1, 2026