SYSTEM Cited by 2 sources

Cloudflare Bot Management

Cloudflare Bot Management is Cloudflare's edge product for classifying incoming traffic as human, good bot (verified), or bad bot, and emitting a bot score used by WAF rules, pay-per-crawl, and managed-rulesets. Stub page on the wiki — the product predates wiki coverage; this stub exists because the 2025-08-04 Perplexity post leans on it as the scoring surface that caught the stealth crawler.

What the 2025-08-04 post discloses

• All undeclared stealth-crawler activity was scored as bot and failed managed challenges — the ML classification worked against Perplexity's stealth UA even after IP + ASN rotation.
• Existing block rules protected customers automatically once the stealth signature shipped — customers didn't need to take action.
• Block signatures for the stealth crawler were added to the managed AI-bots ruleset, available to all customers including the free tier.
• Challenge-instead-of-block is supported — customers who don't want to block can challenge suspect requests, giving real humans a way through.
• Verified Bots directory is the allowlist side of the same product; delisting Perplexity (this post) moves them out of the known-good corpus and subjects their traffic to standard scoring.

The post is the canonical wiki instance of patterns/stealth-crawler-detection-fingerprint — Cloudflare's ML + network-signal fingerprinting producing a stealth-crawler signature that survives rotation of every obvious identifier. See concepts/ml-bot-fingerprinting.

Pipeline position (from systems/pay-per-crawl)

Bot Management runs between WAF and pay-per-crawl on every request:

1. WAF rules.
2. Bot Management — this product.
3. Pay-per-crawl rules engine (if enabled).

A bot scored as malicious never reaches pay-per-crawl.

Seen in

• sources/2025-11-18-cloudflare-outage-on-november-18-2025 — the module whose feature-file generator caused Cloudflare's worst outage since 2019. A ClickHouse permission migration caused the feature-file generator to produce a file with ~doubled rows; the fixed 200-feature preallocated cap in the FL2 bots module was breached; .unwrap() panicked every worker thread; ~3 hours of core-traffic 5xx. FL1 customers got silent bot-score-0 fallback instead of 5xx — rules blocking "bot traffic" saw a flood of false positives. First wiki disclosure that feature files regenerate every 5 minutes and propagate fleet-wide (an instance of the patterns/global-configuration-push antipattern), and that the bots module preallocates memory for features per concepts/preallocated-memory-budget / concepts/feature-file-size-limit.
• sources/2025-08-04-cloudflare-perplexity-stealth-undeclared-crawlers — canonical wiki instance; bot-management + managed-ruleset signatures are the enforcement endpoint for the stealth crawling.

Related

• systems/web-bot-auth — cryptographic-identity primitive that feeds Verified Bots directly.
• systems/pay-per-crawl — downstream consumer of bot score.
• systems/cloudflare-managed-ruleset — managed-ruleset sibling that hosts the stealth-crawler signatures.
• concepts/verified-bots / concepts/ml-bot-fingerprinting / concepts/bot-vs-human-frame.
• patterns/verified-bot-delisting / patterns/stealth-crawler-detection-fingerprint.
• companies/cloudflare.