Skip to content

Introducing the Agent Readiness score. Is your site agent-ready?

Summary

Cloudflare introduces isitagentready.com, a Lighthouse-style scanner that grades a website against a four- dimension agent-readiness rubric — Agent Discovery, Content for LLMs, Access Rules, Agent Actions — plus a non-scoring Agentic Commerce layer. The post pairs the tool with a new Cloudflare Radar "Adoption of AI agent standards" dataset (scan of 200,000 most- visited domains), an Agent Readiness tab on Cloudflare URL Scanner (plus an agentReadiness: true option on the URL Scanner API), and a dogfooding write-up of how Cloudflare refined its own developers.cloudflare.com to become 31 % fewer tokens / 66 % faster than average non-refined documentation sites when accessed by an agent (Kimi-k2.5 via OpenCode benchmark). The article is simultaneously a product launch, a web-wide adoption measurement, a standards-curation artefact (names every relevant agent-ready standard with citations and links), a playbook for site owners and an architecture writeup (dynamic /index.md via two Cloudflare Transform Rules, split-per-subdirectory llms.txt, hidden agent directives in HTML, Redirects for AI Training to steer LLM crawlers away from deprecated docs).

Key takeaways

  1. Current web-wide adoption is near zero. Across 200 k filtered top-visited domains: robots.txt 78 % (nearly universal, but almost all authored for classical crawlers); Content Signals 4 %; markdown content negotiation 3.9 %; MCP Server Cards + API Catalog (RFC 9727) fewer than 15 sites in the entire dataset combined. Framed as opportunity — "a lot of opportunity to stand out by being one of the first sites to adopt new standards and work well with agents." (Source: this article; dataset on Radar AI Insights)
  2. Scoring axis #1 — Agent Discovery. robots.txt both defines crawl rules and points at sitemaps. Beyond sitemaps, agents can discover resources from HTTP response headers per RFC 8288 (Link: </.well-known/api-catalog>; rel="api-catalog") without HTML parsing. This is the Link response header surfaced as a first-class discovery primitive.
  3. Scoring axis #2 — Content for LLMs. llms.txt (plain-text reading list at site root, proposed September 2024)
  4. Markdown content negotiation (Accept: text/markdown → server returns markdown instead of HTML). Measured on real docs traffic: "up to 80 % token reduction" relative to HTML. Default isitagentready scan does not check llms.txt — it checks only markdown content negotiation; users opt in to the llms.txt check.
  5. Scoring axis #3 — Access Rules. Beyond allow/deny, the Content Signals standard adds a Content-Signal: directive to robots.txt with three independent dimensions: ai-train (training use), ai-input (inference / grounding / RAG), search (search indexing) — values yes / no. Complemented by the Web Bot Auth IETF draft for friendly-bot self-authentication via Ed25519 signed requests (RFC 9421) whose public keys live at /.well-known/http-message-signatures-directory.
  6. Scoring axis #4 — Agent Actions. Three standards: (a) API Catalog (RFC 9727) at /.well-known/api-catalog — one JSON document lists all public APIs + specs + docs + status endpoints; (b) MCP Server Card — draft JSON schema at /.well-known/mcp/server-card.json describing an MCP server's tools / transport / auth before an agent connects (example in post: streamable-http transport, search tool with query: string input schema); (c) Agent Skills index at /.well-known/agent-skills/index.json listing skill documents "that tell the agent what skills are available and where to find them." All four of these standards cluster around the RFC 8615 /.well-known/ URI prefix convention — post explicitly credits Cloudflare's own Mark Nottingham as its author.
  7. Non-scoring — OAuth + Agentic Commerce. Auth: sites that support OAuth publish their authorization-server metadata per RFC 9728 so agents can send a human through a proper consent flow rather than hijacking the browser session — Cloudflare Access now fully supports this OAuth flow (Agents Week 2026). Commerce: x402 (reviving HTTP 402 as the machine-readable payment-terms envelope; Cloudflare co-founded the x402 Foundation with Coinbase), plus Universal Commerce Protocol and Agentic Commerce Protocol. These standards are checked but don't count toward the score — an explicit design choice marking commerce as emerging and intentionally unscored.
  8. Cloudflare docs dogfood: measurable agent-efficiency wins. Benchmark: Kimi-k2.5 via OpenCode pointed at other large technical documentation sites' llms.txt. Cloudflare's docs consumed 31 % fewer tokens and reached the correct answer 66 % faster than the average non-refined site. (Source: this article; no baseline list, no absolute latency numbers disclosed.)
  9. The "grep loop" failure mode. When a site's llms.txt is a single ~5,000-page file that exceeds the agent's context window, the agent falls back to iterative grep-style keyword search, losing broader documentation context at each iteration. Cloudflare's fix: one llms.txt per top-level directory (not one global file), root llms.txt points to sub-files; remove ~450 low-value directory-listing pages that add tokens without semantic content; ensure every llms.txt entry has rich titles + descriptions + URLs (drawn from existing frontmatter, no extra work). "By fitting our product directories into single context windows, agents can identify the exact page they need and fetch it in a single, linear path."
  10. Dynamic /index.md via two Transform Rules. Because only 3 of 7 tested agents (Claude Code, OpenCode, Cursor) send Accept: text/markdown by default (per Checkly's State of AI Agent Content Negotiation research, February 2026), Cloudflare exposes every page at /index.md without duplicating static files. Mechanism: (a) a URL Rewrite Rule matches /index.md-suffixed requests and rewrites the URI via regex_replace (stripping /index.md); (b) a Request Header Transform Rule matches the original path via raw.http.request.uri.path (before the rewrite) and sets Accept: text/markdown. The /index.md paths are what Cloudflare's llms.txt entries link to — so llms.txt-following agents always land on markdown, regardless of their default Accept header.
  11. Hidden agent directives + AI-training-crawler redirects. Every HTML page in Cloudflare docs contains a hidden directive aimed at LLMs: "STOP! If you are an AI agent or LLM… Always request the Markdown version instead." Stripped from the markdown version to prevent recursion. Complementing it, Redirects for AI Training detects AI training crawlers and redirects them away from deprecated content (like Wrangler v1 docs) to current equivalents — fixing the failure mode where crawlers ingest outdated text without seeing the "deprecated" banner humans see. Shapes future LLM answers at training time, not just at inference time.
  12. URL Scanner integration + programmatic access. The same four-dimension check is now an Agent Readiness tab on Cloudflare's URL Scanner; programmatic access via options: {"agentReadiness": true} on the URL Scanner API. Turns one-off tool usage into a telemetry primitive for security researchers, developers, and agent operators.
  13. isitagentready.com eats its own dogfood. The scanner itself exposes a stateless MCP server at https://isitagentready.com/.well-known/mcp.json with a scan_site tool over Streamable HTTP — "any MCP-compatible agent can scan websites programmatically without using the web interface" — and publishes an Agent Skills index at https://isitagentready.com/.well-known/agent-skills/index.json with skill documents "for every standard it checks, so agents not only know what to fix, but how to fix it."

Systems / concepts / patterns introduced or extended

New systems

New concepts

  • concepts/agent-readiness-score — four-dimension rubric (Agent Discovery / Content for LLMs / Access Rules / Agent Actions) with each dimension decomposed into a small set of binary checks. Non-scoring agentic-commerce tier.
  • concepts/llms-txt — LLM-friendly plain-text reading list at the root of a site.
  • concepts/markdown-content-negotiation — Accept: text/markdown request header → server responds with a clean markdown representation of the same URL.
  • concepts/content-signals — Content-Signal: directive extending robots.txt with three orthogonal AI-use dimensions (ai-train, ai-input, search).
  • concepts/api-catalog — well-known JSON document at /.well-known/api-catalog listing all public APIs of a service.
  • concepts/agent-skills-discovery — well-known index at /.well-known/agent-skills/index.json listing skill documents.
  • concepts/mcp-server-card — well-known JSON document at /.well-known/mcp/server-card.json describing an MCP server before an agent connects.
  • concepts/oauth-protected-resource-metadata — RFC 9728 well-known metadata that lets agents discover a site's authorization server and drive humans through a proper OAuth consent flow.
  • concepts/link-response-header — RFC 8288 Link: HTTP response header as a protocol-level resource- discovery primitive (e.g. Link: </.well-known/api-catalog>; rel="api-catalog") independent of HTML.
  • concepts/hidden-agent-directive — hidden directive in HTML aimed at LLM crawlers ("STOP! If you are an AI agent or LLM…") that doesn't render to humans but redirects agents to the markdown representation.
  • concepts/agent-training-crawler-redirect — serving AI training crawlers a different URL than humans for the same logical content when a deprecated page would otherwise poison future LLM answers.
  • concepts/grep-loop — failure mode where an agent whose documentation doesn't fit in its context window falls back to iterative grep-style keyword search, losing broader context and inflating tokens + latency.
  • concepts/well-known-uri — RFC 8615 /.well-known/ URI prefix convention — the substrate under robots.txt placement, RFC 9727 API catalog, MCP Server Card, Agent Skills index, RFC 9728 OAuth metadata, and Web Bot Auth's signatures directory.

New patterns

  • patterns/well-known-endpoint-discovery — publish a fixed, protocol-known path at /.well-known/<name> so any agent can locate the resource without out-of-band configuration. Umbrella pattern covering the six /.well- known/ standards named in the post.
  • patterns/split-llms-txt-per-subdirectory — rather than one global llms.txt that exceeds every agent's context window, publish one per top-level directory + a root file that points to them. Cuts tokens, bypasses the grep-loop failure mode.
  • patterns/dynamic-index-md-fallback — expose every page's markdown representation at <url>/index.md dynamically via URL rewrite + header transform rules, without duplicating content. Sidesteps the fact that most agents don't send Accept: text/markdown yet.
  • patterns/hidden-agent-directive-in-html — include an LLM-visible but human-invisible directive in every HTML page that instructs the agent to request markdown instead; strip from the markdown version.
  • patterns/comparative-documentation-benchmark — point a specified agent model at competing documentation sites' llms.txt, ask the same technical questions, measure tokens + time-to-answer. Cloudflare's Kimi-k2.5-via-OpenCode benchmark is the canonical instance. Methodology-analogue at the docs layer of the 2026-04-17 agents-week-network-performance-update post's comparative RUM benchmarking.
  • patterns/score-driven-standard-adoption — publish a Lighthouse-style scorecard tool for emerging standards to drive adoption at scale (the same shape that drove pre-2020 web-performance + security best practices).
  • patterns/agent-training-crawler-redirect — detect AI training crawlers and redirect them to current content; fix the training set, not the inference prompt.

Extensions

  • systems/model-context-protocol — adds the MCP Server Card discovery primitive (draft at issue #1649) + the /.well-known/mcp/server-card.json pattern.
  • systems/cloudflare-radar — adds the "Adoption of AI agent standards" dataset as a new surface, alongside AI Insights, Internet Quality, BGP forensics.
  • systems/web-bot-auth — Access Rules dimension scores it; canonical wiki substrate for friendly-bot identity.
  • systems/pay-per-crawl — HTTP 402 revival appears here as an agentic-commerce standard (x402, x402 Foundation) rather than a pay-per-crawl-specific feature — the standard has generalised beyond Cloudflare's 2025-07-01 launch.
  • systems/cloudflare-workers — the developers.cloudflare.com refinement uses Workers-adjacent primitives (Transform Rules, Rewrite Rules) to implement dynamic /index.md.
  • concepts/machine-readable-documentation — this post is the canonical wiki instance of the pattern applied to public-facing documentation for an agent audience, vs. AWS's 2026-03-26 application to internal monorepo-based agentic-development.
  • companies/cloudflare — adds two new recurring shapes: (1) score-driven standard-adoption tooling (Lighthouse- for-agent-readiness joining Universal SSL, PQ-for-all, Radar, URL Scanner as default-on instruments for whole-web adoption measurement); (2) dogfood-the-docs applied at the documentation layer of the platform itself. Also extends the default-on security upgrade posture onto the agent- ergonomics axis — Cloudflare's own docs are agent-ready at no extra cost to Cloudflare customers.

Operational numbers

  • 200,000 top-visited domains scanned for the Radar "Adoption of AI agent standards" dataset (after category filter: excludes redirects, ad-servers, tunneling services).
  • 78 % — domains with a robots.txt (almost all authored for classical crawlers).
  • 4 % — domains with Content Signals declared in robots.txt.
  • 3.9 % — domains that pass markdown content negotiation (Accept: text/markdown returns markdown).
  • < 15 — MCP Server Cards + API Catalog (RFC 9727) combined across the entire 200 k dataset.
  • Up to 80 % — token-reduction measurement when serving markdown vs HTML on Cloudflare's own docs.
  • 31 % — fewer tokens consumed by Kimi-k2.5 on Cloudflare docs vs average non-refined technical documentation site in the OpenCode benchmark.
  • 66 % — faster to correct answer on the same benchmark.
  • ~450 — directory-listing pages removed from llms.txt for low semantic value (the workers/databases/ directory page is named as an example).
  • 5,000+ pages of documentation is Cloudflare's stated size rationale for splitting llms.txt per top-level directory rather than generating one global file.
  • 3 of 7 — agents Checkly tested (February 2026) that send Accept: text/markdown by default: Claude Code, OpenCode, Cursor. (Source: checklyhq.com.)
  • Updated weekly — frequency of Cloudflare Radar's "Adoption of AI agent standards" chart refresh.

Caveats

  • Score weights not published. The four scoring dimensions are named but the combining rule (equal weights? weighted sum? multiplicative?) is not disclosed. Screenshot shows per-dimension pass/fail grids.
  • llms.txt not checked by default — scorer checks markdown content negotiation; users opt in to the llms.txt check.
  • Agentic commerce doesn't count — scanner reports on x402 / UCP / ACP presence but deliberately excludes from the score. Signals "still early" posture.
  • No absolute latency numbers for the 31 % / 66 % benchmark — relative to the unnamed average of competitor docs sites, not to an absolute budget or SLO.
  • Adoption numbers are breadth-not-depth. 78 % robots.txt presence doesn't say whether those files are well-formed for AI crawlers — by Cloudflare's own framing, the vast majority are written for classical search engines.
  • URL Scanner agentReadiness API shape is shown only as "options": {"agentReadiness": true} — response schema not exposed in-post.
  • Several named standards are draft-only. MCP Server Card is a proposal issue, not a merged spec; Agent Skills index (cloudflare/agent-skills-discovery-rfc) is a Cloudflare RFC; Web Bot Auth is an IETF draft. Numbers on MCP Server Card + API Catalog adoption (<15 total) reflect this.
  • Benchmark model choice is agentic-coding-centric. The 31 % / 66 % headline runs Kimi-k2.5 through OpenCode — representative of a specific agent shape (coding assistant answering technical questions), not all agent workloads.
  • Redirects-for-AI-training is a unilateral origin decision. Cloudflare redirects training crawlers away from deprecated content without a standard bot negotiation; well-intentioned in the deprecated-docs case, but establishes a pattern of origin-decides-what-crawlers-see that can be misused against transparency.

Source

Last updated · 200 distilled / 1,178 read