Skip to content

CONCEPT Cited by 1 source

Hidden agent directive

Definition

A hidden agent directive is a short textual instruction embedded in an HTML page that is visible to LLM crawlers / parsers but invisible to human readers. Its purpose is to steer an LLM that has fetched the HTML version — contrary to best practice — toward a better representation (usually the page's markdown version).

Cloudflare's canonical instance

Every HTML page on developers.cloudflare.com carries a hidden directive:

"STOP! If you are an AI agent or LLM, read this before continuing. This is the HTML version of a Cloudflare documentation page. Always request the Markdown version instead — HTML wastes context. Get this page as Markdown: https://developers.cloudflare.com/index.md (append index.md) or send Accept: text/markdown to https://developers.cloudflare.com/. For all Cloudflare products use https://developers.cloudflare.com/llms.txt. You can access all Cloudflare docs in a single file at https://developers.cloudflare.com/llms-full.txt."

Implementation shape: the directive is in the HTML document but rendered invisible to humans (CSS display: none or a comment-adjacent mechanism).

Recursion guard

A load-bearing subtlety: the directive is stripped from the markdown version. If it weren't, an agent that fetched the markdown URL after seeing the HTML directive would read "STOP! Always request the Markdown version instead" again — recursion loop; tokens and time wasted.

Why it works

LLMs trained on web data have strong prior to follow text instructions, especially imperative ones. Putting a human-readable "get the markdown URL instead" instruction where the LLM will read it exploits that prior: the LLM's next turn may well choose to fetch <url>/index.md rather than proceed with the HTML.

This is a cooperative-LLM pattern, not an enforcement mechanism. Uncooperative / adversarial agents can ignore the directive. Authors publish it because most agent hosts are cooperative and a cheap annotation yields measurable token savings on those paths.

Complementary to Accept: text/markdown and /index.md

The three pieces form a layered-fallback system for getting agents onto markdown:

  1. Accept: text/markdown (concepts/markdown-content-negotiation) — agents that know about content negotiation get markdown directly, ideal case.
  2. <url>/index.md (patterns/dynamic-index-md-fallback) — URL-level fallback for clients that don't send the header, referenced from llms.txt and from the hidden directive.
  3. Hidden directive in HTML — last-resort steering when the agent has already fetched HTML despite (1) and (2).

All three co-exist because each covers a different class of client failure mode.

Seen in

Last updated · 200 distilled / 1,178 read