PATTERN Cited by 1 source

Conservative broker headroom

Definition

Treat unused broker capacity as a reliability feature, not waste. Size Kafka brokers by their per-broker peak ingress, egress, EBS throughput, CPU, and remote-copy headroom — not just aggregate cluster averages. Keep sustained CPU below a conservative threshold so background work (replication, tiered storage offload, consumer fan-out) can continue during traffic spikes.

Key principles (Atlassian StreamHub)

1. Choose broker instances for network and EBS headroom, not just CPU utilization.
2. Scale out in AZ multiples for even distribution.
3. Balance partitions and leaders continuously — hot partitions break clusters before total capacity is exhausted.
4. Use producer/consumer quotas to protect the shared broker network budget from noisy neighbors.

Why conventional utilization targets fail at streaming scale

Kafka traffic is amplified: producer ingress is only one part. Replication (RF=3 means 2× extra writes), consumer fan-out, cross-region relay, retries, and tiered-storage remote copy all compete for broker resources. A broker that looks 50% utilized on average may be at 90%+ during a 30-second spike — and background workers stall.

Seen in

• sources/2026-07-28-atlassian-scaling-streamhub-transitioning-from-kinesis-to-kafka — "We stopped treating unused broker capacity as waste. For a system this critical, headroom is a reliability feature."