Skip to content

PATTERN Cited by 1 source

Signed Bot Request

Signed Bot Request is the design pattern for giving an automated client (crawler / bot / agent) a cryptographic identity that an origin can verify per-request, replacing IP allowlists / reverse-DNS / API keys as the mechanism. Canonical implementation is Cloudflare's Web Bot Auth; canonical consumer is pay-per-crawl (which requires cryptographic identity before it can bill).

The three-ingredient recipe

  1. Ed25519 keypair per bot operator. Smallest / fastest of the modern asymmetric signature schemes — ~64-byte signatures, sub-millisecond sign/verify, constant-time implementations widely available.
  2. JWK-formatted public key at a hosted directory URL. Bot operator publishes its current public key (optionally many, via keyid) in JWK (RFC 7517) format at a well-known URL they control. Directory URL becomes the trust anchor for the bot's identity.
  3. Per-request RFC 9421 signature over the bot's private key, with signature-agent header pointing at the JWK directory, keyid selecting the key, alg=ed25519, created/expires for replay window, random nonce, and a tag disambiguating the deployment (e.g. web-bot-auth).

The three headers on every request:

Signature-Agent: "https://signature-agent.example.com"
Signature-Input: sig2=(…) ;keyid=… ;alg=ed25519 ;created=… ;expires=… ;nonce=… ;tag="web-bot-auth"
Signature: sig2=:<base64url>:

Verifier logic

  1. Parse Signature-Agent → fetch JWK directory (cached per origin).
  2. Look up key by keyid.
  3. Canonicalize covered fields per RFC 9421.
  4. Verify Ed25519 signature.
  5. Check timestamp window + nonce freshness.
  6. Match tag against what this deployment expects — prevents cross-deployment signature replay.

Load-bearing invariant: the signed request must cryptographically bind to the same keypair that the registered bot operator publishes — exactly the patterns/identity-to-key-binding check shared with OPKSSH's PK Token verification and other public-key-commitment designs.

Why it beats its predecessors

Scheme Attack surface
IP allowlist Shared egress IPs, cloud-NAT fluidity, on-host malware
Reverse-DNS (Googlebot style) DNS hijack, RDNS-forge, slow to propagate
User-Agent string Trivially forged
Bearer API key Leaks once → lasts forever; rotation costly
Signed Bot Request Requires private-key compromise + in-window replay + nonce collision

No shared secret. No server-side per-bot state. Key rotation via JWK directory. Verification happens stateless on the receive side.

Scales to marketplace

Because verification is stateless and the trust anchor is a web-hosted directory, the pattern composes naturally with marketplace-style aggregation (patterns/merchant-of-record-aggregation). Cloudflare doesn't need bilateral contracts with every bot operator; it registers (directory-URL + UA) pairs and lets verification flow from there.

Seen in

Last updated · 200 distilled / 1,178 read