---
title: "Bytespider: block or allow this AI crawler · DotsAgent"
description: "Bytespider is run by ByteDance and listed under training. Its robots.txt token, whether it follows the rules, its IP list and how to control it."
url: https://dotsagent.io/agent-ready/bots/bytespider
---

ByteDance · Training

# What is Bytespider?

ByteDance's crawler, listed for training in community registries. ByteDance has no official page for it.

- **robots.txt token**: `Bytespider`
- **Operator**: ByteDance
- **Purpose**: Training
- **Obeys robots.txt**: The operator says robots.txt may not apply, or community reports say it ignores it.
- **Documented by operator**: No. The token is known only from community lists such as ai.robots.txt.

Community reports say it does not respect robots.txt.

## Block Bytespider from your whole site

`robots.txt`

```
User-agent: Bytespider
Disallow: /
```

## Allow it, but keep private paths out

`robots.txt`

```
User-agent: Bytespider
Allow: /
Disallow: /admin/
```

[Add it to a robots.txt in the generator →](https://dotsagent.io/agent-ready/robots-txt)

## Sources

1. [github.com](https://github.com/ai-robots-txt/ai.robots.txt)/ai-robots-txt/ai.robots.txt

Independent reference for people who build AI agents. Not affiliated with any vendor named here.

© 2026 DotsAgent · Facts checked October 1, 2026