Skip to content

SYSTEM Cited by 4 sources

Web Bot Auth

Web Bot Auth is Cloudflare's proposal for cryptographically identifying bots / crawlers to origins, replacing IP-allowlist-based bot verification with per-request Ed25519 signatures over HTTP Message Signatures (RFC 9421). It is the identity substrate under Cloudflare's pay-per-crawl feature (2025-07-01): because crawlers pay per request, the origin must be certain which crawler is asking, and shared egress IPs make IP-based identity trivially spoofable.

Enrollment flow

A bot operator onboards by:

  1. Generating an Ed25519 keypair. Ed25519 is the modern default — ~64-byte signatures, constant-time verification, ~ms signing on commodity hardware.
  2. Publishing the public key at a hosted directory, in JWK (JSON Web Key, RFC 7517) format. The directory URL is what identifies the bot to origins.
  3. Registering the directory URL + user-agent string with Cloudflare.
  4. Configuring the crawler to sign every request with RFC 9421 HTTP Message Signatures using the private key.

Once accepted, every crawler request carries signature-agent, signature-input, and signature request headers and is verifiable by any Cloudflare edge node.

Request shape

GET /example.html
Signature-Agent: "https://signature-agent.example.com"
Signature-Input: sig2=("@authority" "signature-agent")
 ;created=1735689600
 ;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U"
 ;alg="ed25519"
 ;expires=1735693200
 ;nonce="e8N7S2MFd/qrd6T2R3tdfAuuANngKI7LFtKYI/vowzk4lAZYadIX6wW25MwG7DCT9RUKAJ0qVkU0mEeLElW1qg=="
 ;tag="web-bot-auth"
Signature: sig2=:jdq0SqOwHdyHr9+r5jw3iYZH6aNGKijYp/EstF4RQTQdi5N5YYKrD+mCT1HA1nZDsi6nJKuHxUi/5Syp3rLWBA==:

Header roles:

  • signature-agent — directory URL; identifies the bot operator and tells the verifier where to fetch keys.
  • signature-input — covered-field list (@authority and signature-agent in the canonical example), keyid selecting which key from the JWK directory, alg="ed25519", created / expires timestamps for replay protection, random nonce, tag field set to web-bot-auth to disambiguate from other RFC 9421 deployments.
  • signature — the Ed25519 signature bytes (base64url).

Verification

On the edge, Cloudflare:

  1. Resolves the signature-agent URL to fetch the JWK directory.
  2. Selects the key matching keyid.
  3. Verifies the Ed25519 signature over the canonicalized covered fields per RFC 9421.
  4. Checks timestamp window (created ≤ now ≤ expires) and nonce freshness.
  5. Emits a verified-crawler identity that downstream features (pay-per-crawl being the first consumer) can use for pricing, billing, policy.

The trust anchor is the JWK directory URL: Cloudflare effectively treats the bot operator's hosted key material as authoritative for the identity claim, the same way OIDC treats an Identity Provider's /.well-known/jwks.json as authoritative for user identity. The load-bearing invariant is identity-to-key-binding — the keyid/signature binds the request to a specific private key, and the registered directory binds that key to a named bot operator.

Positioning vs. prior bot-verification schemes

  • IP allowlists — shared egress IPs and NAT pools make IP impersonation cheap; insufficient for "charge this crawler" decisions.
  • Reverse-DNS verification (classic Googlebot model) — still IP-based + DNS-based; subject to DNS-cache / hijack attacks and cloud egress fluidity.
  • Custom user-agent tokens / API keys — trivially spoofable; key rotation hard; no per-request cryptographic evidence.
  • Web Bot Auth — per-request cryptographic signature over a published public key; no shared secret; anyone can verify; keys rotate via JWK directory; no custom protocol (rides inside RFC 9421 signatures that HTTP clients/servers are already adopting for other uses).

Seen in

  • sources/2025-07-01-cloudflare-pay-per-crawl — pay-per-crawl relies on Web Bot Auth as its identity substrate; this is the wiki's canonical instance.
  • sources/2026-04-17-cloudflare-introducing-the-agent-readiness-score-is-your-site-agent-ready — Web Bot Auth is the scored Access Rules check in the Agent Readiness Score. Public-key directory path is /.well-known/http-message-signatures-directory; the isitagentready scanner fetches it to verify a site hosts a bot-key directory. Part of the /.well-known/ cluster of agent-ergonomic web standards — see well-known-endpoint-discovery.
  • sources/2026-04-21-cloudflare-moving-past-bots-vs-humans — positions Web Bot Auth as the identity branch of the post-bot-vs-human architecture for web protection. The population Web Bot Auth targets — search crawlers, cloud platforms, AI training pipelines — tolerates attribution because predictable access is worth the cost. The anonymous branch (serving distributed low-volume clients — humans, AI assistants acting on human behalf, researchers) is covered by Privacy Pass / ARC / ACT. Both branches use active, cryptographically-bound signals (per-request signatures) — the displacement in 2026 is from passive-signal inference (IP allowlists, TLS fingerprinting), not away from cryptography.
  • sources/2025-08-04-cloudflare-perplexity-stealth-undeclared-crawlers — the enforcement precedent for the identity branch. Cloudflare's controlled brand-new-domain experiment caught Perplexity running an undeclared stealth crawler to evade origin-side blocks of its declared crawlers; Cloudflare de-listed Perplexity from Verified Bots (verified-bot-delisting) and shipped ML stealth-signature blocks in its managed AI-bots ruleset. Positive control in the same experiment: ChatGPT-User fetched robots.txt, honored Disallow, and stopped — plus ChatGPT Agent signs via Web Bot Auth. The post operationalizes Web Bot Auth as the "what cooperative crawlers should do" side of a two-sided enforcement program.
  • systems/pay-per-crawl — first consumer of Web Bot Auth; the identity it produces gates billing events.
  • systems/privacy-pass — anonymous-branch sibling; serves the client population that needs anonymity while still proving behavior.
  • http-message-signatures — the RFC 9421 primitive Web Bot Auth builds on.
  • verified-bots — the broader category of identifying legitimate bots to origins.
  • bot-vs-human-frame — why the bot-vs-human binary fails; the identity / anonymous branching replaces it.
  • patterns/signed-bot-request — the request-shape pattern (Ed25519 key + JWK directory + per-request signature).
  • identity-to-key-binding — generic verifier invariant; shared with OPKSSH's PK Token check.
  • anonymous-attribute-proof — the anonymous-branch counterpart pattern.
  • Cloudflare Web Bot Auth docs

Seen in

Last updated · 766 distilled / 2,225 read