[{"data":1,"prerenderedAt":238},["ShallowReactive",2],{"copy:en:site:":3,"copy:en:agentweb:hub,ui":122,"copy:en:site:common":236},{"common":4,"nav":16,"footer":44,"home":51,"error":115},{"skip":5,"menu":6,"language":7,"sections":8,"footer":9,"tool":10,"home":11,"breadcrumbs":12,"copy":13,"copied":14,"download":15},"Skip to content","Menu","Language","Sections","Footer","tool","Home","Breadcrumbs","Copy","Copied","Download",{"groups":17,"items":25},{"build":18,"connect":19,"instruct":20,"run":21,"web":22,"secure":23,"track":24},"Build","Connect","Instruct","Run","Open your site","Secure","Track",{"models":26,"cost":27,"frameworks":28,"mcp":29,"mcpConfig":30,"mcpClients":31,"agentsMd":32,"files":33,"openclaw":34,"agentReady":35,"llmsTxt":36,"robotsTxt":37,"apiCatalog":38,"bots":39,"security":40,"incidents":41,"checklist":42,"changelog":43},"Models and API prices","Agent cost calculator","Frameworks and SDKs","MCP servers","MCP config generator","MCP clients","AGENTS.md generator","Instruction files","Self-host OpenClaw","Agent-ready websites","llms.txt generator","robots.txt for AI bots","API catalog generator","AI bot directory","Agent security","Incident log","Security checklist","Changelog",{"about":45,"privacy":46,"terms":47,"sitemap":48,"disclaimer":49,"checked":50},"About","Privacy","Terms","Sitemap","Independent reference for people who build AI agents. Not affiliated with any vendor named here.","Facts checked {date}",{"seo":52,"eyebrow":55,"title":56,"lead":57,"ctaMcp":58,"ctaCost":59,"ctaAgents":60,"sheetLabel":61,"sheetTitle":62,"sheet":63,"sheetNote":69,"sectionsTitle":70,"sections":71,"latestTitle":98,"allChanges":99,"rulesTitle":100,"rules":101,"aboutLink":114},{"title":53,"description":54},"DotsAgent: reference and tools for building AI agents","LLM API prices, an agent cost calculator, MCP configs for 13 clients, AGENTS.md and llms.txt generators, and agent security notes. Free, in 18 languages.","For developers building AI agents","The agent builder's desk reference","Prices, protocols, config files and security notes for agent work, each checked against the vendor's own docs. Look a fact up, generate the file you need and get back to your code.","Generate an MCP config","Price an agent task","Write AGENTS.md","Current figures","State of the stack",{"mcp":64,"openclaw":65,"cheapest":66,"models":67,"servers":29,"bots":68},"MCP spec","OpenClaw","Cheapest model, $\u002FM in\u002Fout","Models priced","AI bots tracked","Each figure is checked by hand against vendor docs.","What's on the site",{"models":72,"mcp":76,"instructions":79,"frameworks":82,"openclaw":85,"agentReady":88,"security":91,"changelog":94},{"title":73,"body":74,"unit":75},"Models and prices","Input, output and cached token prices per million, side by side, with a calculator for whole agent runs.","models",{"title":29,"body":77,"unit":78},"A catalogue of Model Context Protocol servers and a config generator that writes the right file for each client.","servers",{"title":33,"body":80,"unit":81},"Write an AGENTS.md and see how CLAUDE.md, .cursor\u002Frules and similar files are read by each tool.","file formats",{"title":28,"body":83,"unit":84},"Compare agent frameworks and vendor SDKs by language, licence, MCP support and multi-agent features.","frameworks",{"title":34,"body":86,"unit":87},"Install, configure and update OpenClaw on your own machine or server, step by step.","current release",{"title":35,"body":89,"unit":90},"Generate llms.txt, robots.txt rules for AI crawlers and an api-catalog, and look up any bot by user agent.","bots listed",{"title":40,"body":92,"unit":93},"The OWASP risks for agents, a log of real incidents and a checklist you can tick off before launch.","incidents logged",{"title":95,"body":96,"unit":97},"API changelog","Dated changes to model APIs, SDKs and protocols, each linked to the vendor's announcement.","entries","Latest changes","All changes","How we keep it accurate",[102,105,108,111],{"title":103,"body":104},"Sourced facts.","Every price, version and flag links to the vendor page it came from, so you can check it yourself.",{"title":106,"body":107},"Dated checks.","Each page shows when its facts were last checked, and stale entries are rechecked or removed.",{"title":109,"body":110},"No paid placement.","Nobody pays to be listed, ranked higher or described more kindly.",{"title":112,"body":113},"Tools run in your browser.","Generators and calculators work locally, and what you type is not sent to a server.","About DotsAgent",{"title":116,"body":117,"home":118,"popular":119,"failed":120,"failedBody":121},"Page not found","This address doesn't match any page. It may have moved, or the link may have a typo.","Go to the home page","Popular tools","Something went wrong","The page failed to load on our side. Try again in a minute, or start from the home page.",{"hub":123,"ui":227},{"seo":124,"eyebrow":127,"title":128,"lead":129,"tools":130,"layersTitle":142,"layersLead":143,"layers":144,"unnamedTitle":208,"unnamed":209,"faqTitle":210,"faq":211},{"title":125,"description":126},"Agent-ready websites: llms.txt, robots.txt, AI bots","Generate llms.txt, a robots.txt for AI bots and an RFC 9727 API catalog, then look up 40 AI crawlers and agents by token, purpose and published IP list.","Open your site · reference","Make your site readable by AI agents","AI crawlers and agents read your site differently from people. The tools below generate the files they look for, and the reference after them says which conventions are standards, which are drafts and how many clients actually use them.",{"llms":131,"robots":134,"catalog":136,"bots":139},{"kind":132,"title":36,"body":133},"Generator","Enter a name, a summary and sections of links, and copy a file in the llmstxt.org format.",{"kind":132,"title":37,"body":135},"Set a policy per bot purpose, flip single crawlers, add Content-Signal and download the result.",{"kind":132,"title":137,"body":138},"API catalog","List your APIs with their OpenAPI files and docs, and get an RFC 9727 linkset plus an nginx block to serve it.",{"kind":140,"title":39,"body":141},"Directory","40 crawlers and agents with their tokens, purposes, robots.txt behaviour and published IP lists.","The conventions, and how settled each one is","Status as of October 2026. Each entry links to the spec or to the evidence behind it, so you can judge what is worth the effort on your site.",[145,155,164,174,185,191,200],{"title":146,"status":147,"statusTone":148,"body":149,"code":150,"codeLabel":151,"sources":152},"robots.txt","Standard","ok","The file at your site root names crawlers by token and lists the paths each one should skip; it is standardised as RFC 9309. AI operators now publish separate tokens for training, search and user-triggered fetches, so a single file can refuse training and still welcome search. Compliance is voluntary, and several user-triggered fetchers say the rules may not apply to them.","User-agent: GPTBot\nDisallow: \u002F\n\nUser-agent: OAI-SearchBot\nAllow: \u002F","\u002Frobots.txt",[153,154],"https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fbots","https:\u002F\u002Fdevelopers.google.com\u002Fsearch\u002Fdocs\u002Fcrawling-indexing\u002Fgoogle-common-crawlers",{"title":156,"status":157,"statusTone":158,"body":159,"code":160,"codeLabel":151,"sources":161},"Content-Signal","Widely used","blue","One extra robots.txt line that states how fetched content may be used: search, ai-input and ai-train, each set to yes or no. Cloudflare released it as a CC0 policy in September 2025 and serves it on about 3.8 million domains through its managed robots.txt. The matching IETF draft expired on 4 April 2026, while the IETF AIPREF working group carries on with a vocabulary draft.","User-agent: *\nContent-Signal: ai-train=no, search=yes, ai-input=yes\nAllow: \u002F",[162,163],"https:\u002F\u002Fblog.cloudflare.com\u002Fcontent-signals-policy\u002F","https:\u002F\u002Fdatatracker.ietf.org\u002Fwg\u002Faipref\u002Fdocuments\u002F",{"title":165,"status":166,"statusTone":167,"body":168,"code":169,"codeLabel":170,"sources":171},"llms.txt","Proposal","warn","A Markdown file at \u002Fllms.txt with an H1 name, a blockquote summary and H2 sections of links, so an agent can reach your key pages without parsing navigation. The spec at llmstxt.org was revised on 10 August 2026, and llms-full.txt is not part of it. We have no operator statement that any major crawler reads the file.","# Acme API\n\n> Card payments for small online shops.\n\n## Docs\n\n- [Quickstart](https:\u002F\u002Fexample.com\u002Fdocs\u002Fquickstart.md): first request","\u002Fllms.txt",[172,173],"https:\u002F\u002Fllmstxt.org\u002F","https:\u002F\u002Fllmstxt.org\u002Fchanges.html",{"title":175,"status":176,"statusTone":177,"body":178,"code":179,"codeLabel":180,"sources":181},"Markdown versions of pages","Partial adoption","signal","Publish a Markdown copy of each page at page.md or page.html.md, or serve Markdown to any request whose Accept header asks for text\u002Fmarkdown. In Checkly's February 2026 test, Claude Code, Cursor and OpenCode sent that header, while Codex, Gemini CLI, GitHub Copilot and Windsurf did not. AnswerShare counted 1,277,965 requests from frontier crawlers and none of them asked for Markdown.","GET \u002Fdocs\u002Fquickstart HTTP\u002F1.1\nHost: example.com\nAccept: text\u002Fmarkdown","Request",[182,183,184],"https:\u002F\u002Fdevelopers.cloudflare.com\u002Ffundamentals\u002Freference\u002Fmarkdown-for-agents\u002F","https:\u002F\u002Fwww.checklyhq.com\u002Fblog\u002Fstate-of-ai-agent-content-negotation\u002F","https:\u002F\u002Fanswershare.com\u002Fmarkdowndoesnotwork",{"title":137,"status":147,"statusTone":148,"body":186,"code":187,"codeLabel":188,"sources":189},"RFC 9727, published in June 2025, reserves \u002F.well-known\u002Fapi-catalog for a list of your public APIs, each linked to its machine-readable description, docs and status page. The response must be application\u002Flinkset+json, and other pages can point to it with the api-catalog link relation.","{\n  \"linkset\": [\n    {\n      \"anchor\": \"https:\u002F\u002Fapi.example.com\u002Fv1\",\n      \"service-desc\": [{ \"href\": \"https:\u002F\u002Fapi.example.com\u002Fv1\u002Fopenapi.json\" }]\n    }\n  ]\n}","\u002F.well-known\u002Fapi-catalog",[190],"https:\u002F\u002Fwww.rfc-editor.org\u002Frfc\u002Frfc9727.html",{"title":192,"status":193,"statusTone":158,"body":194,"code":195,"codeLabel":196,"sources":197},"Signed agent requests (Web Bot Auth)","Draft","Agents sign each request with HTTP Message Signatures (RFC 9421) and name their operator in a Signature-Agent header. You check the signature against the keys that operator publishes under \u002F.well-known\u002Fhttp-message-signatures-directory. The IETF working group draft dates from 1 September 2026; ChatGPT agent already signs as https:\u002F\u002Fchatgpt.com, and Google is experimenting with https:\u002F\u002Fagent.bot.goog.","Signature-Agent: \"https:\u002F\u002Fchatgpt.com\"","Request header",[198,199],"https:\u002F\u002Fdatatracker.ietf.org\u002Fdoc\u002Fdraft-ietf-webbotauth-httpsig-protocol\u002F","https:\u002F\u002Fblog.cloudflare.com\u002Fsigned-agents\u002F",{"title":201,"status":202,"statusTone":203,"body":204,"sources":205},"NLWeb","Stalled","","Microsoft's NLWeb gives a site \u002Fask and \u002Fmcp endpoints that agents can query in natural language, and its spec is at version 0.55. Lumar reports that the nlweb.ai certificate expired on 4 August 2026 and that the repository has been quiet since June, so wait before building on it.",[206,207],"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FNLWeb\u002Fblob\u002Fmain\u002Fdocs\u002Fnlweb-rest-api.md","https:\u002F\u002Fagentic-readiness.lumar.io\u002Fdocs\u002Fcapabilities\u002Fnl-web","Agents that don't announce themselves","Some agents publish no robots.txt token and browse as the user would, so no User-agent rule can reach them. ChatGPT agent signs its requests with Web Bot Auth and sends Signature-Agent: \"https:\u002F\u002Fchatgpt.com\", which you can check against OpenAI's key directory; Cloudflare labels that traffic chatgpt-agent. Gemini Spark in local Chrome mode and Meta's Muse have no documented signal at all.","Questions about agent-ready sites",[212,215,218,221,224],{"q":213,"a":214},"What makes a website agent-ready?","No single standard defines it. In practice it means a robots.txt that names AI bots by purpose, optional files such as llms.txt and an API catalog, Markdown versions of key pages, and a way to verify the agents you let in. Of these, only robots.txt and the API catalog are published RFCs.",{"q":216,"a":217},"Do AI crawlers read llms.txt?","We have no operator documentation saying that any crawler in our directory fetches it. Lighthouse checks whether the file exists, and an agent you point at it can read it like any other page. Treat it as a convenience for agents, not a search feature.",{"q":219,"a":220},"Should I block AI crawlers on my site?","Decide per purpose rather than all at once. Training tokens such as GPTBot, ClaudeBot and Google-Extended can be refused without touching search. Blocking search tokens such as OAI-SearchBot or PerplexityBot keeps you out of those products entirely.",{"q":222,"a":223},"Can I block ChatGPT agent with robots.txt?","No, because it publishes no token. You can recognise it by its Web Bot Auth signature from https:\u002F\u002Fchatgpt.com and allow or deny it at your server or CDN. A valid signature proves which operator sent the request, not who the user is or what they may do.",{"q":225,"a":226},"Is Content-Signal an official standard?","No. It is a CC0 policy from Cloudflare, launched in September 2025 and served on about 3.8 million domains through Cloudflare's managed robots.txt. Its IETF draft expired on 4 April 2026, and the AIPREF working group is still drafting a shared vocabulary.",{"crumb":228,"purposes":229,"sources":235},"Agent-ready sites",{"training":230,"search":231,"user":232,"agent":233,"ads":234},"Training","Search","User fetches","Agents","Ads","Sources",{"common":237},{"skip":5,"menu":6,"language":7,"sections":8,"footer":9,"tool":10,"home":11,"breadcrumbs":12,"copy":13,"copied":14,"download":15},1790863351024]