Skip to content

SYSTEM Cited by 4 sources

Web Bot Auth

Web Bot Auth is Cloudflare's proposal for cryptographically identifying bots / crawlers to origins, replacing IP-allowlist-based bot verification with per-request Ed25519 signatures over HTTP Message Signatures (RFC 9421). It is the identity substrate under Cloudflare's pay-per-crawl feature (2025-07-01): because crawlers pay per request, the origin must be certain which crawler is asking, and shared egress IPs make IP-based identity trivially spoofable.

Enrollment flow

A bot operator onboards by:

  1. Generating an Ed25519 keypair. Ed25519 is the modern default — ~64-byte signatures, constant-time verification, ~ms signing on commodity hardware.
  2. Publishing the public key at a hosted directory, in JWK (JSON Web Key, RFC 7517) format. The directory URL is what identifies the bot to origins.
  3. Registering the directory URL + user-agent string with Cloudflare.
  4. Configuring the crawler to sign every request with RFC 9421 HTTP Message Signatures using the private key.

Once accepted, every crawler request carries signature-agent, signature-input, and signature request headers and is verifiable by any Cloudflare edge node.

Request shape

GET /example.html
Signature-Agent: "https://signature-agent.example.com"
Signature-Input: sig2=("@authority" "signature-agent")
 ;created=1735689600
 ;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U"
 ;alg="ed25519"
 ;expires=1735693200
 ;nonce="e8N7S2MFd/qrd6T2R3tdfAuuANngKI7LFtKYI/vowzk4lAZYadIX6wW25MwG7DCT9RUKAJ0qVkU0mEeLElW1qg=="
 ;tag="web-bot-auth"
Signature: sig2=:jdq0SqOwHdyHr9+r5jw3iYZH6aNGKijYp/EstF4RQTQdi5N5YYKrD+mCT1HA1nZDsi6nJKuHxUi/5Syp3rLWBA==:

Header roles:

  • signature-agent — directory URL; identifies the bot operator and tells the verifier where to fetch keys.
  • signature-input — covered-field list (@authority and signature-agent in the canonical example), keyid selecting which key from the JWK directory, alg="ed25519", created / expires timestamps for replay protection, random nonce, tag field set to web-bot-auth to disambiguate from other RFC 9421 deployments.
  • signature — the Ed25519 signature bytes (base64url).

Verification

On the edge, Cloudflare:

  1. Resolves the signature-agent URL to fetch the JWK directory.
  2. Selects the key matching keyid.
  3. Verifies the Ed25519 signature over the canonicalized covered fields per RFC 9421.
  4. Checks timestamp window (created ≤ now ≤ expires) and nonce freshness.
  5. Emits a verified-crawler identity that downstream features (pay-per-crawl being the first consumer) can use for pricing, billing, policy.

The trust anchor is the JWK directory URL: Cloudflare effectively treats the bot operator's hosted key material as authoritative for the identity claim, the same way OIDC treats an Identity Provider's /.well-known/jwks.json as authoritative for user identity. The load-bearing invariant is patterns/identity-to-key-binding — the keyid/signature binds the request to a specific private key, and the registered directory binds that key to a named bot operator.

Positioning vs. prior bot-verification schemes

  • IP allowlists — shared egress IPs and NAT pools make IP impersonation cheap; insufficient for "charge this crawler" decisions.
  • Reverse-DNS verification (classic Googlebot model) — still IP-based + DNS-based; subject to DNS-cache / hijack attacks and cloud egress fluidity.
  • Custom user-agent tokens / API keys — trivially spoofable; key rotation hard; no per-request cryptographic evidence.
  • Web Bot Auth — per-request cryptographic signature over a published public key; no shared secret; anyone can verify; keys rotate via JWK directory; no custom protocol (rides inside RFC 9421 signatures that HTTP clients/servers are already adopting for other uses).

Seen in

Last updated · 200 distilled / 1,178 read