SYSTEM Cited by 1 source
Atlassian StreamHub¶
What it is¶
StreamHub is Atlassian's central event streaming platform — the real-time data backbone powering clicks, impressions, API calls, CDC events, and telemetry across the entire product suite (Jira, Confluence, Bitbucket, Trello, etc.). It ingests and processes 150 billion events per day and delivers over 225 billion events per day, averaging 1.68 million events/sec with peaks exceeding 3.2 million events/sec.
Architecture evolution¶
StreamHub evolved through three phases:
- Amazon Kinesis era (up to ~22B events/day): Zero-ops managed streaming. Hit cost, reliability, and retention limits at 5× growth.
- AWS MSK migration (current): Apache Kafka via MSK with Kafka Tiered Storage (5-minute local EBS hot tier + 7-day S3 cold tier). Small platform team focuses on stream architecture and client libraries.
- Hardened MSK (current): Multi-shard clusters with failover runbooks, companion regions, ingress rate limiting, quarantine paths, and client quotas.
Key design decisions¶
- Tiered Storage separates cost (S3 for historical data) from performance (EBS for real-time consumers).
- Cluster sharding limits blast radius so incidents affect a subset of traffic.
- Failover cluster escape hatch provides recovery when the managed control plane is unavailable.
- Ingress rate limiting + quarantine protects brokers from upstream spikes and malformed traffic.
- Client quotas (
producer_byte_rate,consumer_byte_rate) prevent noisy-neighbor monopolization. - Companion regions enable data-residency-compliant DR.