[{"data":1,"prerenderedAt":190},["ShallowReactive",2],{"copy:en:site:":3,"copy:en:agentweb:bots,ui":122,"copy:en:site:common":188},{"common":4,"nav":16,"footer":44,"home":51,"error":115},{"skip":5,"menu":6,"language":7,"sections":8,"footer":9,"tool":10,"home":11,"breadcrumbs":12,"copy":13,"copied":14,"download":15},"Skip to content","Menu","Language","Sections","Footer","tool","Home","Breadcrumbs","Copy","Copied","Download",{"groups":17,"items":25},{"build":18,"connect":19,"instruct":20,"run":21,"web":22,"secure":23,"track":24},"Build","Connect","Instruct","Run","Open your site","Secure","Track",{"models":26,"cost":27,"frameworks":28,"mcp":29,"mcpConfig":30,"mcpClients":31,"agentsMd":32,"files":33,"openclaw":34,"agentReady":35,"llmsTxt":36,"robotsTxt":37,"apiCatalog":38,"bots":39,"security":40,"incidents":41,"checklist":42,"changelog":43},"Models and API prices","Agent cost calculator","Frameworks and SDKs","MCP servers","MCP config generator","MCP clients","AGENTS.md generator","Instruction files","Self-host OpenClaw","Agent-ready websites","llms.txt generator","robots.txt for AI bots","API catalog generator","AI bot directory","Agent security","Incident log","Security checklist","Changelog",{"about":45,"privacy":46,"terms":47,"sitemap":48,"disclaimer":49,"checked":50},"About","Privacy","Terms","Sitemap","Independent reference for people who build AI agents. Not affiliated with any vendor named here.","Facts checked {date}",{"seo":52,"eyebrow":55,"title":56,"lead":57,"ctaMcp":58,"ctaCost":59,"ctaAgents":60,"sheetLabel":61,"sheetTitle":62,"sheet":63,"sheetNote":69,"sectionsTitle":70,"sections":71,"latestTitle":98,"allChanges":99,"rulesTitle":100,"rules":101,"aboutLink":114},{"title":53,"description":54},"DotsAgent: reference and tools for building AI agents","LLM API prices, an agent cost calculator, MCP configs for 13 clients, AGENTS.md and llms.txt generators, and agent security notes. Free, in 18 languages.","For developers building AI agents","The agent builder's desk reference","Prices, protocols, config files and security notes for agent work, each checked against the vendor's own docs. Look a fact up, generate the file you need and get back to your code.","Generate an MCP config","Price an agent task","Write AGENTS.md","Current figures","State of the stack",{"mcp":64,"openclaw":65,"cheapest":66,"models":67,"servers":29,"bots":68},"MCP spec","OpenClaw","Cheapest model, $\u002FM in\u002Fout","Models priced","AI bots tracked","Each figure is checked by hand against vendor docs.","What's on the site",{"models":72,"mcp":76,"instructions":79,"frameworks":82,"openclaw":85,"agentReady":88,"security":91,"changelog":94},{"title":73,"body":74,"unit":75},"Models and prices","Input, output and cached token prices per million, side by side, with a calculator for whole agent runs.","models",{"title":29,"body":77,"unit":78},"A catalogue of Model Context Protocol servers and a config generator that writes the right file for each client.","servers",{"title":33,"body":80,"unit":81},"Write an AGENTS.md and see how CLAUDE.md, .cursor\u002Frules and similar files are read by each tool.","file formats",{"title":28,"body":83,"unit":84},"Compare agent frameworks and vendor SDKs by language, licence, MCP support and multi-agent features.","frameworks",{"title":34,"body":86,"unit":87},"Install, configure and update OpenClaw on your own machine or server, step by step.","current release",{"title":35,"body":89,"unit":90},"Generate llms.txt, robots.txt rules for AI crawlers and an api-catalog, and look up any bot by user agent.","bots listed",{"title":40,"body":92,"unit":93},"The OWASP risks for agents, a log of real incidents and a checklist you can tick off before launch.","incidents logged",{"title":95,"body":96,"unit":97},"API changelog","Dated changes to model APIs, SDKs and protocols, each linked to the vendor's announcement.","entries","Latest changes","All changes","How we keep it accurate",[102,105,108,111],{"title":103,"body":104},"Sourced facts.","Every price, version and flag links to the vendor page it came from, so you can check it yourself.",{"title":106,"body":107},"Dated checks.","Each page shows when its facts were last checked, and stale entries are rechecked or removed.",{"title":109,"body":110},"No paid placement.","Nobody pays to be listed, ranked higher or described more kindly.",{"title":112,"body":113},"Tools run in your browser.","Generators and calculators work locally, and what you type is not sent to a server.","About DotsAgent",{"title":116,"body":117,"home":118,"popular":119,"failed":120,"failedBody":121},"Page not found","This address doesn't match any page. It may have moved, or the link may have a typo.","Go to the home page","Popular tools","Something went wrong","The page failed to load on our side. Try again in a minute, or start from the home page.",{"bots":123,"ui":179},{"seo":124,"crumb":39,"eyebrow":127,"title":128,"lead":129,"purposeFilter":130,"all":131,"search":132,"operator":133,"anyOperator":134,"cols":135,"noToken":141,"marks":142,"signed":146,"ips":147,"empty":148,"purposesTitle":149,"purposeHelp":150,"verifyTitle":156,"verify":157,"toGenerator":161,"faqTitle":162,"faq":163},{"title":125,"description":126},"AI bot directory: 40 crawlers, tokens and IP lists","40 AI crawlers and agents from OpenAI, Anthropic, Google, Apple, Meta, Amazon and others, with robots.txt token, purpose, compliance and IP list.","{n} crawlers and agents","AI crawler and agent directory","Each AI bot we could confirm, with the token to use in robots.txt, what it fetches for, whether its operator says it follows the rules, and how to check a request really comes from it. Tokens known only from the community ai.robots.txt registry are marked as such on their own pages.","Filter by purpose","All","Search by name, token or operator","Operator","Any operator",{"bot":136,"token":137,"purpose":138,"robots":139,"verify":140},"Bot","Token","Purpose","Obeys robots.txt","Verify","No token",{"yes":143,"no":144,"unknown":145},"Yes","No","Not stated","Signed requests","IP list","No bots match these filters.","What the purposes mean",{"training":151,"search":152,"user":153,"agent":154,"ads":155},"Collects pages for model training or general research crawling. Google-Extended and Applebot-Extended are control tokens that never fetch anything themselves.","Indexes pages for a search product, from Google Search to AI answer engines such as ChatGPT search, Perplexity and DuckAssist. Blocking one keeps you out of that product.","Fetches a page because a person asked for it in a chat or tool. Several operators say robots.txt may not apply to these requests.","Browses and acts on a user's behalf, often in a full browser. Many have no token and can be identified only by signature, if at all.","Checks ad landing pages that advertisers submitted. These are left out of the robots.txt generator.","Check a bot is who it says it is",[158,159,160],"A user-agent string costs nothing to fake. Most large operators publish the IP ranges their bots use as JSON files, marked IP list in the table. Compare the request's address with the list for that token, and treat a match on the name alone as unverified.","Some operators also document reverse DNS. Common Crawl, for example, says CCBot hosts resolve to *.crawl.commoncrawl.org; confirm that the hostname resolves back to the same address before you trust it.","Agents increasingly sign requests with HTTP Message Signatures (RFC 9421) under the Web Bot Auth draft. The request names its operator in Signature-Agent, and you verify it with the public keys at that domain's \u002F.well-known\u002Fhttp-message-signatures-directory. A valid signature proves the operator, not the user behind it or what that user may do.","Build a robots.txt from this list","Questions about AI crawlers",[164,167,170,173,176],{"q":165,"a":166},"Which AI crawlers ignore robots.txt?","By their operators' own documentation, ChatGPT-User, Perplexity-User, Google-Agent, Google-GeminiNotebook, Amzn-User, meta-externalfetcher and facebookexternalhit may not follow it, mostly because a user triggered the fetch. Bytespider is reported by the community not to respect it.",{"q":168,"a":169},"What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?","GPTBot collects pages for training OpenAI models, OAI-SearchBot indexes pages for ChatGPT search, and ChatGPT-User fetches a page when a person asks for it. The first two follow robots.txt; OpenAI says its rules may not apply to the third.",{"q":171,"a":172},"Which AI agents have no robots.txt token?","ChatGPT agent, Gemini Spark and Meta's Muse. ChatGPT agent can be verified through its Web Bot Auth signature from https:\u002F\u002Fchatgpt.com. Gemini Spark in local Chrome mode looks like the user's own browser, and Muse has no published user agent.",{"q":174,"a":175},"How do I verify a request really comes from Googlebot?","Google publishes Googlebot's IP ranges as a JSON file, linked from its entry in this directory. Check the request's address against those ranges rather than trusting the user-agent string.",{"q":177,"a":178},"Where does this list of AI bots come from?","Each entry links to the operator's documentation where it exists, and tokens without it come from the community ai.robots.txt registry. We left out DeepSeekBot, Kimi-User, Manus-User, NovaAct, Devin and PetalBot because we found no operator documentation for them. The list was checked on 1 October 2026.",{"crumb":180,"purposes":181,"sources":187},"Agent-ready sites",{"training":182,"search":183,"user":184,"agent":185,"ads":186},"Training","Search","User fetches","Agents","Ads","Sources",{"common":189},{"skip":5,"menu":6,"language":7,"sections":8,"footer":9,"tool":10,"home":11,"breadcrumbs":12,"copy":13,"copied":14,"download":15},1790863351104]