Skip to content

ATLASSIAN 2026-07-28

Read original ↗

Scaling StreamHub: Transitioning from Kinesis to Kafka for 145 Billion Daily Events

Summary

Atlassian's StreamHub team details their multi-year migration from Amazon Kinesis to AWS MSK (Managed Streaming for Apache Kafka) to handle event streaming at massive scale — growing from 22 billion to 150 billion events per day (1.68M events/sec average, 3.2M events/sec peak). The post covers the architectural reasons for the migration (Kinesis throttling, shard cost, retention limits, EFO costs), the adoption of Kafka Tiered Storage to separate hot/cold data, six critical failure modes encountered at scale with managed Kafka, and the hardening solutions they implemented including sharded clusters, failover runbooks, companion regions, rate limiting, and client quotas.

Key Takeaways

  1. Kinesis hits cost and reliability limits at extreme scale. At 5× growth, Kinesis shards became prohibitively expensive (linear cost scaling), 24-hour retention was insufficient (needed 7 days for consumer replay), and shared egress per shard plus EFO pricing made multi-consumer fan-out economically unviable. StreamHub API reliability needed to go from 99.99% to 99.999%.

  2. Kafka Tiered Storage is a cost strategy, not a capacity strategy. The local tier (last 5 minutes on EBS) serves real-time consumers; the remote tier (S3, 7-day retention) serves historical reads. But if broker CPU/network is saturated, the async remote-copy workers fall behind — local disks fill linearly, and S3 isn't a substitute for broker headroom.

  3. Plan capacity around the hottest broker, not the cluster average. Partition skew and hot leaders cause incidents before total cluster capacity is exhausted. Replication, consumer fan-out, cross-region relay, retries, and tiered storage offload all compete for the same broker network/EBS budget.

  4. Managed-service control planes can become unavailable exactly when you need them. During an AZ degradation, the MSK control plane blocked scaling operations, reboots, and reassignment actions — the recovery levers were locked out at incident time. Data-plane Kafka was running but unmodifiable.

  5. Scaling MSK is slow, asymmetric, and irreversible. Storage increases are permanent (no downsizing), rolling broker upgrades take ~15 min/broker (hours for a large cluster), and adding brokers doesn't help until partitions are rebalanced — rebalancing itself adds load during an incident.

  6. S3 retention changes create API-rate "delete storms." Decreasing tiered storage retention triggers mass deletes that compete with regular remote writes, pausing offload and causing local disk growth — even though S3 has effectively unlimited capacity.

  7. Rate limiting and quarantine are reliability tools, not just cost tools. Moving traffic control upstream (before Kafka) via per-tenant/topic rate limits and quarantine for malformed/failing traffic prevents shared broker resources from being overwhelmed by one bad actor.

  8. Client quotas (producer_byte_rate, consumer_byte_rate) are blast-radius controls. They prevent a single misconfigured pipeline from monopolizing broker network bandwidth across all tenants.

  9. Failover clusters + sharding provide escape hatches. Splitting into multiple shards reduces per-incident blast radius; a pre-provisioned failover cluster (IaC templates, ~2h standup time) lets operators route traffic away from a stuck primary without depending on that cluster's control plane.

  10. Companion regions satisfy data-residency during DR. Pre-approved regional pairings within the same compliance boundary turn disaster recovery from improvised geography-shifting into a pre-vetted, policy-compliant failover path.

Operational Numbers

Metric Value
Daily events ingested 150 billion
Daily events delivered 225 billion
Average throughput 1.68 million events/sec
Peak throughput 3.2 million events/sec
Previous scale (Kinesis era) 22 billion events/day
Local tier retention 5 minutes
Remote tier (S3) retention 7 days
Target API reliability 99.999%
Failover cluster standup time ~2 hours
Shard failover ETA ~15 minutes
Rolling broker upgrade time ~15 min/broker

Systems & Concepts Extracted

Systems: Apache Kafka, AWS MSK, Amazon Kinesis, Amazon S3, Amazon EBS

Concepts: Tiered storage (hot/cold), broker network saturation, partition skew, control-plane/data-plane separation (in failure context), AZ failure modes, managed-service cooldown windows, irreversible scaling operations, client quotas as blast-radius controls

Patterns: Failover cluster escape hatch, ingress rate limiting + quarantine, cluster sharding for blast radius, companion-region compliant failover, conservative broker headroom as reliability investment

Caveats

  • Atlassian is a Tier-3 source; however, this post is exceptionally rich in distributed-systems internals, production incident detail, and scaling trade-offs with concrete numbers.
  • The post does not identify specific MSK broker instance types or exact dollar costs.
  • "Companion regions" are described conceptually but the compliance framework details are not provided.

Source

Last updated · 602 distilled / 1,824 read