Skip to content

SYSTEM Cited by 4 sources

Cloudflare Bot Management

Cloudflare Bot Management is Cloudflare's edge product for classifying incoming traffic as human, good bot (verified), or bad bot, and emitting a bot score used by WAF rules, pay-per-crawl, and managed-rulesets. Stub page on the wiki — the product predates wiki coverage; this stub exists because the 2025-08-04 Perplexity post leans on it as the scoring surface that caught the stealth crawler.

What the 2025-08-04 post discloses

  • All undeclared stealth-crawler activity was scored as bot and failed managed challenges — the ML classification worked against Perplexity's stealth UA even after IP + ASN rotation.
  • Existing block rules protected customers automatically once the stealth signature shipped — customers didn't need to take action.
  • Block signatures for the stealth crawler were added to the managed AI-bots ruleset, available to all customers including the free tier.
  • Challenge-instead-of-block is supported — customers who don't want to block can challenge suspect requests, giving real humans a way through.
  • Verified Bots directory is the allowlist side of the same product; delisting Perplexity (this post) moves them out of the known-good corpus and subjects their traffic to standard scoring.

The post is the canonical wiki instance of patterns/stealth-crawler-detection-fingerprint — Cloudflare's ML + network-signal fingerprinting producing a stealth-crawler signature that survives rotation of every obvious identifier. See concepts/ml-bot-fingerprinting.

Pipeline position (from systems/pay-per-crawl)

Bot Management runs between WAF and pay-per-crawl on every request:

  1. WAF rules.
  2. Bot Management — this product.
  3. Pay-per-crawl rules engine (if enabled).

A bot scored as malicious never reaches pay-per-crawl.

Seen in

  • sources/2026-05-01-cloudflare-code-orange-fail-small-complete — Snapstone-onboarded feature-file pipeline. The Bot Management feature-file generator is one of the canonical workloads Cloudflare's Code Orange programme brought under Snapstone's health-mediated rollout — moving the pipeline from rapid-fleet-wide-push antipattern to staged + health-gated + automated-rollback by default. Complemented at the consumer tier by concepts/fail-stale (keep the last-known-good feature file on validation failure) and Codex-enforced ingest validation on every MR. The post's evaluation: "if the same Bot Management change that caused the failure in November were to roll out now, the system would detect the failure in an early stage of the deployment, before it had affected anything more than a small percentage of traffic."
  • sources/2025-11-18-cloudflare-outage-on-november-18-2025 — the module whose feature-file generator caused Cloudflare's worst outage since 2019. A ClickHouse permission migration caused the feature-file generator to produce a file with ~doubled rows; the fixed 200-feature preallocated cap in the FL2 bots module was breached; .unwrap() panicked every worker thread; ~3 hours of core-traffic 5xx. FL1 customers got silent bot-score-0 fallback instead of 5xx — rules blocking "bot traffic" saw a flood of false positives. First wiki disclosure that feature files regenerate every 5 minutes and propagate fleet-wide (an instance of the patterns/global-configuration-push antipattern), and that the bots module preallocates memory for features per concepts/preallocated-memory-budget / concepts/feature-file-size-limit.
  • sources/2025-08-04-cloudflare-perplexity-stealth-undeclared-crawlers — canonical wiki instance; bot-management + managed-ruleset signatures are the enforcement endpoint for the stealth crawling.

Role in frontier-model defence

In the context of frontier AI cyber models, Bot Management catches probing traffic before models can build a reconnaissance map. It scores every request on automation likelihood using network-wide signals: client behaviour, browser fingerprint, connection patterns. An attack only lands if it can find a soft spot — Bot Management's job is to prevent the mapping phase.

(Source: sources/2026-06-09-cloudflare-defend-against-frontier-cyber-models)

Last updated · 542 distilled / 1,571 read