SYSTEM Cited by 1 source
Cloudflare Security Insights¶
Overview¶
Security Insights is Cloudflare's system for providing actionable security recommendations to every Cloudflare account. It performs automated scans across all accounts, zones, and DNS records, detecting potential security risks and misconfigurations.
Architecture¶
- Scheduler — triggers scans on a per-account and per-zone cadence with adaptive rate limiting (recomputed every 30 minutes).
- Kafka topic — 30 partitions; scheduler publishes scan-trigger messages.
- Go checkers — specialized microservices consuming from Kafka, each scanning specific asset types or configurations. Each checker has its own consumer group.
- Internal API — receives scan results (insights) from checkers; writes to Postgres.
- Postgres — primary in Portland, Oregon; stores all discovered insights.
Scale (post-optimization, 2026)¶
-
120 scans/second sustained throughput (peak scheduling)
- 30 Kafka partitions per checker (unchanged from pre-optimization)
- Up to 500,000 insights written per single API call
- All free-tier accounts scanned automatically
- Scanning frequency doubled for all customers
Key design decisions¶
- Batch-parallel consumption — checkers consume messages in batches, process each in a goroutine (patterns/batch-goroutine-parallel-consumption).
- Fast/slow lane split — two consumer groups per checker to avoid concepts/head-of-line-blocking (patterns/fast-lane-slow-lane-consumer-split).
- Active-passive API — API follows the Postgres primary to avoid cross-region latency (concepts/active-passive-failover).
- Adaptive scheduling — rate limit dynamically adjusts to account/zone population (patterns/adaptive-rate-limited-scheduling).
- Independent zone scheduling — zones have their own
last_scheduled_at, decoupled from their parent account.