[{"data":1,"prerenderedAt":197},["ShallowReactive",2],{"copy:en:site:":3,"copy:en:agentweb:robots,ui":122,"copy:en:site:common":195},{"common":4,"nav":16,"footer":44,"home":51,"error":115},{"skip":5,"menu":6,"language":7,"sections":8,"footer":9,"tool":10,"home":11,"breadcrumbs":12,"copy":13,"copied":14,"download":15},"Skip to content","Menu","Language","Sections","Footer","tool","Home","Breadcrumbs","Copy","Copied","Download",{"groups":17,"items":25},{"build":18,"connect":19,"instruct":20,"run":21,"web":22,"secure":23,"track":24},"Build","Connect","Instruct","Run","Open your site","Secure","Track",{"models":26,"cost":27,"frameworks":28,"mcp":29,"mcpConfig":30,"mcpClients":31,"agentsMd":32,"files":33,"openclaw":34,"agentReady":35,"llmsTxt":36,"robotsTxt":37,"apiCatalog":38,"bots":39,"security":40,"incidents":41,"checklist":42,"changelog":43},"Models and API prices","Agent cost calculator","Frameworks and SDKs","MCP servers","MCP config generator","MCP clients","AGENTS.md generator","Instruction files","Self-host OpenClaw","Agent-ready websites","llms.txt generator","robots.txt for AI bots","API catalog generator","AI bot directory","Agent security","Incident log","Security checklist","Changelog",{"about":45,"privacy":46,"terms":47,"sitemap":48,"disclaimer":49,"checked":50},"About","Privacy","Terms","Sitemap","Independent reference for people who build AI agents. Not affiliated with any vendor named here.","Facts checked {date}",{"seo":52,"eyebrow":55,"title":56,"lead":57,"ctaMcp":58,"ctaCost":59,"ctaAgents":60,"sheetLabel":61,"sheetTitle":62,"sheet":63,"sheetNote":69,"sectionsTitle":70,"sections":71,"latestTitle":98,"allChanges":99,"rulesTitle":100,"rules":101,"aboutLink":114},{"title":53,"description":54},"DotsAgent: reference and tools for building AI agents","LLM API prices, an agent cost calculator, MCP configs for 13 clients, AGENTS.md and llms.txt generators, and agent security notes. Free, in 18 languages.","For developers building AI agents","The agent builder's desk reference","Prices, protocols, config files and security notes for agent work, each checked against the vendor's own docs. Look a fact up, generate the file you need and get back to your code.","Generate an MCP config","Price an agent task","Write AGENTS.md","Current figures","State of the stack",{"mcp":64,"openclaw":65,"cheapest":66,"models":67,"servers":29,"bots":68},"MCP spec","OpenClaw","Cheapest model, $\u002FM in\u002Fout","Models priced","AI bots tracked","Each figure is checked by hand against vendor docs.","What's on the site",{"models":72,"mcp":76,"instructions":79,"frameworks":82,"openclaw":85,"agentReady":88,"security":91,"changelog":94},{"title":73,"body":74,"unit":75},"Models and prices","Input, output and cached token prices per million, side by side, with a calculator for whole agent runs.","models",{"title":29,"body":77,"unit":78},"A catalogue of Model Context Protocol servers and a config generator that writes the right file for each client.","servers",{"title":33,"body":80,"unit":81},"Write an AGENTS.md and see how CLAUDE.md, .cursor\u002Frules and similar files are read by each tool.","file formats",{"title":28,"body":83,"unit":84},"Compare agent frameworks and vendor SDKs by language, licence, MCP support and multi-agent features.","frameworks",{"title":34,"body":86,"unit":87},"Install, configure and update OpenClaw on your own machine or server, step by step.","current release",{"title":35,"body":89,"unit":90},"Generate llms.txt, robots.txt rules for AI crawlers and an api-catalog, and look up any bot by user agent.","bots listed",{"title":40,"body":92,"unit":93},"The OWASP risks for agents, a log of real incidents and a checklist you can tick off before launch.","incidents logged",{"title":95,"body":96,"unit":97},"API changelog","Dated changes to model APIs, SDKs and protocols, each linked to the vendor's announcement.","entries","Latest changes","All changes","How we keep it accurate",[102,105,108,111],{"title":103,"body":104},"Sourced facts.","Every price, version and flag links to the vendor page it came from, so you can check it yourself.",{"title":106,"body":107},"Dated checks.","Each page shows when its facts were last checked, and stale entries are rechecked or removed.",{"title":109,"body":110},"No paid placement.","Nobody pays to be listed, ranked higher or described more kindly.",{"title":112,"body":113},"Tools run in your browser.","Generators and calculators work locally, and what you type is not sent to a server.","About DotsAgent",{"title":116,"body":117,"home":118,"popular":119,"failed":120,"failedBody":121},"Page not found","This address doesn't match any page. It may have moved, or the link may have a typo.","Go to the home page","Popular tools","Something went wrong","The page failed to load on our side. Try again in a minute, or start from the home page.",{"robots":123,"ui":186},{"seo":124,"crumb":37,"eyebrow":127,"title":128,"lead":129,"form":130,"presets":139,"purposeNotes":143,"comments":148,"signalNote":152,"limits":153,"botsLink":154,"explainTitle":155,"explain":156,"faqTitle":169,"faq":170},{"title":125,"description":126},"robots.txt generator for AI crawlers and agents","Build a robots.txt that refuses AI training but keeps search: 35 bot tokens grouped by purpose, Content-Signal lines, private paths and a sitemap line.","Open your site · generator","robots.txt generator for AI bots","Pick a policy for each kind of AI bot, override single crawlers and copy a file that follows RFC 9309. Training, search and user-triggered fetchers use separate tokens, so you can refuse one and welcome another.",{"preset":131,"allow":132,"block":133,"tapHint":134,"unset":135,"disallow":136,"disallowHint":137,"sitemap":138},"Start from","Allow","Block","Tap a bot to flip it against its group's setting.","unset","Private paths","One path per line. Allowed bots and all other crawlers are kept out of these.","Sitemap URL",{"open":140,"noTraining":141,"closed":142},"Allow everything","Block training only","Block all AI bots",{"training":144,"search":145,"user":146,"agent":147},"Blocking these opts your pages out of model training by these operators and leaves your search listings alone.","Blocking these also blocks Googlebot and Applebot, which takes you out of Google Search and Apple's Siri, Spotlight and Safari results.","These fetch a page because a person asked for it, and several operators say robots.txt may not apply, so a block here is a request rather than a guarantee.","Google-Agent generally ignores robots.txt and most browsing agents have no token at all, so this group changes little in practice.",{"blocked":149,"allowed":150,"everyone":151},"AI crawlers blocked from the whole site","Allowed AI crawlers, private paths excluded","All other crawlers","Content-Signal says how fetched pages may be used: search for search results, ai-input for feeding AI answers and ai-train for model training. It comes from Cloudflare's CC0 policy and states a preference without blocking anything.","robots.txt is voluntary. ChatGPT-User, Perplexity-User and other user-triggered fetchers say the rules may not apply to them, and some agents have no token at all. To enforce a block, match requests against published IP lists or signatures and deny them at your firewall or WAF.","See every bot in the directory","How the generated file works",[157,160,163,166],{"title":158,"body":159},"A crawler follows its own group","A crawler that finds its token in a User-agent line follows that group and skips the rules under User-agent: *. That is why the generator repeats your private paths for the bots you allow by name, and gives blocked bots a single Disallow: \u002F.",{"title":161,"body":162},"Some tokens never crawl","Google-Extended and Applebot-Extended have no user agent of their own. Googlebot and Applebot do the fetching, and the extended tokens tell Google and Apple whether that content may train their models; Google also applies Google-Extended to Gemini grounding.",{"title":164,"body":165},"Training and search are split by most operators","OpenAI, Anthropic, Amazon and Mistral each run separate tokens for training and for search, so the No training preset keeps you in their search products. Meta is the exception: meta-externalagent covers both training and indexing.",{"title":167,"body":168},"Applebot falls back to Googlebot","If your file has no Applebot group, Apple says Applebot follows your Googlebot rules instead. Applebot also ignores Crawl-delay, as does Amazonbot, while Anthropic says ClaudeBot honours it.","Questions about robots.txt and AI bots",[171,174,177,180,183],{"q":172,"a":173},"How do I block GPTBot but stay in ChatGPT search?","Disallow GPTBot and leave OAI-SearchBot allowed. GPTBot collects training data, OAI-SearchBot indexes pages for ChatGPT search, and ChatGPT-User handles fetches a person asks for. OpenAI documents all three on one page, each with its own IP list.",{"q":175,"a":176},"Does blocking Google-Extended remove my site from Google Search?","Google-Extended is a control token with no user agent of its own; Googlebot does the crawling. Disallowing it tells Google not to use your content for Gemini training and grounding, while your Googlebot rules decide what is crawled for Search.",{"q":178,"a":179},"Do AI crawlers respect Crawl-delay?","Some do. Anthropic says ClaudeBot honours it, while Apple and Amazon say Applebot and Amazonbot do not. Rate limiting at your server is the dependable option.",{"q":181,"a":182},"Why does ChatGPT-User still visit after I blocked it?","OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person asked for the page. Perplexity-User, Amzn-User and meta-externalfetcher come with similar caveats. Where the operator publishes an IP list, match requests against it and block at your firewall.",{"q":184,"a":185},"What does Content-Signal do in robots.txt?","It states how content may be used once fetched, through three signals: search, ai-input and ai-train, each set to yes or no. It expresses a preference and blocks nothing. Cloudflare's Markdown for Agents adds ai-train=yes, search=yes, ai-input=yes by default, so check its output if you use that feature.",{"crumb":187,"purposes":188,"sources":194},"Agent-ready sites",{"training":189,"search":190,"user":191,"agent":192,"ads":193},"Training","Search","User fetches","Agents","Ads","Sources",{"common":196},{"skip":5,"menu":6,"language":7,"sections":8,"footer":9,"tool":10,"home":11,"breadcrumbs":12,"copy":13,"copied":14,"download":15},1790863351067]