Skip to content

SYSTEM Cited by 3 sources

Pay Per Crawl

Pay Per Crawl is a Cloudflare feature (announced 2025-07-01 in private beta) that lets publishers monetize AI-crawler access to their content at internet scale. For each request to a participating site, the domain owner can choose one of three outcomes per crawler — Allow (free), Charge (flat per-request price returned as HTTP 402 Payment Required), or Block (no content, no pricing) — enforced by a rules engine that runs after the site's WAF and bot-management layer.

Crawlers are identified cryptographically via Web Bot Auth: an Ed25519 keypair, a hosted JWK directory, and per-request RFC 9421 HTTP Message Signatures. Price negotiation uses new HTTP request/response headers (crawler-price, crawler-max-price, crawler-exact-price, crawler-charged) in two flows ( reactive-discovery and preemptive-price). Cloudflare is the Merchant of Record — it aggregates billing events, charges crawlers, and distributes earnings to publishers (patterns/merchant-of-record-aggregation).

Pipeline position

On every crawler request the Cloudflare edge executes in order:

  1. WAF rules (existing).
  2. Bot management / bot-blocking (existing).
  3. Pay-per-crawl rules engine — applies the Allow / Charge / Block decision after the security layers. Pay-per-crawl never overrides a WAF block; publishers keep their existing security posture and layer monetization on top.

Publisher controls

  • Domain-wide flat price at launch — single per-request USD value applied uniformly across the entire zone.
  • Per-crawler override: publisher can bypass the charge for a specific crawler (free pass for a partner / an indexer whose crawl is desired / a pre-negotiated out-of-band contract).
  • Charge-for-no-billing-relationship is the functional equivalent of 403 Forbidden — no content is returned — but the crawler is told pricing exists, converting silent blocks into standing offers for a future billing relationship.
  • Not at launch: per-path prices, per-content-type prices, dynamic / demand-based pricing, granular licenses (training vs inference vs search).

Protocol surface

Request headers (from crawler)

  • signature-agent, signature-input, signature — Web Bot Auth / RFC 9421 (required on every pay-per-crawl-aware request).
  • crawler-max-price: USD XX.XX — preemptive; declares the crawler's ceiling. Served 200 OK with crawler-charged if configured price ≤ max. If configured price > max, returns 402.
  • crawler-exact-price: USD XX.XX — reactive; used after receiving a 402 with a crawler-price, confirms willingness to pay exactly that price.
  • Only one of crawler-max-price / crawler-exact-price per request.

Response headers (from Cloudflare)

  • HTTP 402 Payment Required + crawler-price: USD XX.XX — pricing advertisement when the crawler didn't declare payment intent or declared a ceiling below the configured price.
  • HTTP 200 OK + crawler-charged: USD XX.XX + server: cloudflare — content served, billing event emitted (crawler is charged the configured price, not the ceiling they declared).

Billing / settlement

Billing events fire only when all of the following are true:

  • the request was signed per Web Bot Auth,
  • the request declared payment intent (crawler-exact-price or crawler-max-price),
  • the response was HTTP 200-family with a crawler-charged header.

Cloudflare aggregates events across publishers and crawlers, charges each crawler in bulk, distributes earnings to each publisher. A single intermediary (Merchant of Record) turns what used to be N × M bilateral contracts into a marketplace.

Intended end-state: agentic paywall

Cloudflare explicitly frames pay-per-crawl as groundwork for an agentic paywall — a future where AI research agents given a user budget hit 402s programmatically, consult the budget, and decide per-resource whether to pay. The HTTP-402 choice and the header-negotiation shape are deliberately designed to generalize from crawler-to-publisher to agent-to-resource.

Seen in

  • sources/2025-07-01-cloudflare-pay-per-crawl — the launch post (Cloudflare Blog, private beta, HN 569).
  • sources/2026-04-17-cloudflare-introducing-the-agent-readiness-score-is-your-site-agent-ready — pay-per-crawl's HTTP-402 primitive evolves into the x402 open standard (Cloudflare co-founded the x402 Foundation with Coinbase). The 2026 Agent Readiness Score scanner checks for x402 + UCP + ACP presence under a non-scoring Agentic Commerce layer — signals the space is emerging and intentionally unscored. Pay-per-crawl is the first-party Cloudflare product that popularised the shape; x402 is the ecosystem standardisation.
  • sources/2025-08-04-cloudflare-perplexity-stealth-undeclared-crawlers — the enforcement precondition. Pay-per-crawl charges verifiable crawler identities; crawlers that evade identity via stealth crawling route around pay-per-crawl entirely. This post documents Cloudflare's non-cooperative-crawler enforcement stack — ML fingerprinting + verified-bot delisting + managed AI-bots ruleset block signatures — which the pay-per-crawl economics require to function. Without it, a publisher charging for access would see "Allow"-priced declared crawlers paying while stealth crawlers fetch the same content for free. The 2025-08-04 post is the enforcement half of what the 2025-07-01 pay-per-crawl post was the monetization half of.
Last updated · 200 distilled / 1,178 read