Skip to content

PATTERN Cited by 1 source

Dynamic /index.md fallback

Pattern

Expose every page's markdown representation at <page-url>/index.md, implemented dynamically via HTTP rewrite rules rather than by duplicating content. Provides a URL-based fallback for agents that don't send Accept: text/markdown — which is still the majority of agents in 2026.

Motivation

Markdown content negotiation is the clean way to serve markdown to agents, but per Checkly's State of AI Agent Content Negotiation research (February 2026), only 3 of 7 tested agents send Accept: text/markdown by default (Claude Code, OpenCode, Cursor). The other four fetch HTML and waste tokens.

Pointing those agents at a distinct URL — <page>/index.md — that always returns markdown sidesteps the header-support gap. The llms.txt entries and hidden HTML directives can then point at the markdown URL rather than rely on header negotiation.

Cloudflare's two-Transform-Rules implementation

Cloudflare implements this without duplicating any static content, using two Cloudflare Transform Rules:

  1. URL Rewrite Rule matches any request with a path ending in /index.md and rewrites the URI (via regex_replace) to strip the /index.md suffix — so the rewritten request hits the same origin route as the human-facing page.
  2. Request Header Transform Rule matches the original request path via raw.http.request.uri.path (crucially, before the rewrite is applied) — if the original path ended in /index.md, the rule sets Accept: text/markdown on the rewritten request.

Net effect: origin server receives a normal-looking request to <page> with Accept: text/markdown; it returns markdown; the client experiences <page>/index.md as a stable markdown URL. No content duplication. No build-step changes. Everything is rule-config at the edge.

Why this is load-bearing at scale

  • ~450 directory-listing pages removed in Cloudflare's llms.txt; the remaining entries all point at /index.md URLs so the agent that follows them gets markdown in a single hop.
  • Caches once per variant. The edge caches the markdown response under the /index.md URL; subsequent agent requests hit cache.
  • Works even when the agent isn't aware of content negotiation at all. The URL itself encodes "I want markdown".

Integration with other patterns

Alternatives and trade-offs

  • Static .md file twins. Every page has a page.md sibling; build step generates them. Works but doubles build output + needs pipeline updates on every content change.
  • Subdomain split (md.example.com/page). Works but requires DNS + cert management and splits analytics / cache / observability stacks.
  • Just send Vary: Accept with content-negotiated response. Cleanest but invisible to clients that don't send the header.

The Transform-Rules variant wins for operational simplicity on Cloudflare — no build changes, no second hostname, no origin- server code changes, cache works automatically.

Portability

The pattern generalises to any HTTP stack with URL-rewrite capability:

  • Nginx: rewrite ^(.*)/index\.md$ $1 last; + an Accept: text/markdown override via proxy_set_header / add_header at the matched location.
  • Apache: mod_rewrite + RequestHeader set Accept.
  • Any CDN with rewrite + header modification (Fastly VCL, AWS CloudFront + Lambda@Edge, Akamai, etc.).

The Cloudflare instance is load-bearing because it's production-proven and tied to a measured 31 % / 66 % agent-benchmark improvement.

Seen in

Last updated · 200 distilled / 1,178 read