Diffbot · Search
What is Diffbot?
Diffbot's crawler for data extraction and its knowledge graph. The token comes from community lists, not Diffbot's own docs.
- robots.txt token
Diffbot- Operator
- Diffbot
- Purpose
- Search
- Obeys robots.txt
- Not stated The operator does not say.
- Documented by operator
- No. The token is known only from community lists such as ai.robots.txt.
Community sources say whether it honours robots.txt depends on Diffbot's own users.
Block Diffbot from your whole site
User-agent: Diffbot
Disallow: /
Allow it, but keep private paths out
User-agent: Diffbot
Allow: /
Disallow: /admin/
Add it to a robots.txt in the generator →