Skip to content

ATLASSIAN

Inside the Events Rail: How Atlassian Delivers 10 Billion Webhooks a Month

Summary

Atlassian's webhooks-processor service — internally called the "events rail" — is the multi-tenant infrastructure that evaluates, matches, and delivers events from Atlassian products (Jira, Confluence, Bitbucket) to Forge apps and customer-controlled webhook endpoints. Handling over 10 billion events/month (~6,000 events/sec sustained, peaks above 600M events/day), the system is built around a two-stage pipeline (Event Matching → Event Delivery) separated by queues, with a layered multi-tenant fairness model that uses negative-lookup caching, queue isolation, per-tenant rate limiting, and per-recipient concurrency limiting to ensure predictable delivery under arbitrary load.

Key takeaways

  1. Selective pub-sub inverts the optimization target. When most events have no matching subscriber, the no-match path is the hot path. Atlassian optimizes it with a negative-lookup cache that stores "no subscriptions" answers with short TTL, substantially reducing registry calls for high-volume event types (Source: sources/2026-07-29-atlassian-inside-the-events-rail).

  2. Two-stage pipeline separates CPU-bound matching from network-bound delivery. Event Matching runs subscription registry lookups and filter-expression evaluation (CPU-intensive, cache-dominated). Event Delivery opens connections to external endpoints with wildly varying latency (network-bound, concurrency-dominated). Splitting them allows independent scaling on the relevant dimension (Source: sources/2026-07-29-atlassian-inside-the-events-rail).

  3. Queues between stages provide flow control as a first-class signal. Queue depth drives autoscaling; permit-based pull from delivery pods means slow deliveries naturally throttle consumption; the queue absorbs matched events during delivery slowdowns — converting invisible latency into a measurable, actionable metric (Source: sources/2026-07-29-atlassian-inside-the-events-rail).

  4. Multi-tenant fairness is layered, not centralized — four mechanisms at four levels. (a) Negative-lookup cache catches evaluation cost; (b) Queue isolation catches head-of-line blocking between pipelines; (c) Per-tenant rate limiting catches capacity exhaustion from noisy tenants; (d) Per-recipient concurrency limiting catches slow-but-not-noisy recipients that never trip rate limits (Source: sources/2026-07-29-atlassian-inside-the-events-rail).

  5. Per-surface configuration model avoids surface-specific branching. Each event surface (Forge product events, Bitbucket hooks, audit logs, workflow events) declares its own destination lookup, routing, retry policy, and failure behaviour against shared core interfaces. Adding a new surface is a self-contained change (Source: sources/2026-07-29-atlassian-inside-the-events-rail).

  6. Backpressure follows TCP congestion-control shape at the application layer. When a downstream shows sustained errors or latency, the system reduces sending rate; when signals recover, it opens gradually. "Pull back fast, recover slow" prevents delivery storms (Source: sources/2026-07-29-atlassian-inside-the-events-rail).

  7. Circuit breakers wrap every upstream dependency. Extension lookup, destination resolution, and the rate-limiter service are each wrapped so a stall in one degrades to scoped failure (delayed delivery on affected pipeline) rather than cascading into system-wide delivery stall (Source: sources/2026-07-29-atlassian-inside-the-events-rail).

  8. Idempotency is delivery-native. Every webhook carries a stable identifier (per event-instance, not per attempt) so recipients can deduplicate across retries. Retries follow exponential backoff with jitter, capped at a per-surface attempt count, then move to a per-pipeline dead-letter queue (Source: sources/2026-07-29-atlassian-inside-the-events-rail).

  9. Growth at ~18% month-over-month, compounding. Two structural drivers: AI agents inside Atlassian products (machine-paced, bursty event generation) and Data Center → Cloud migrations (step-change per-tenant volume on migration days). The layered architecture was built to absorb these without rebuilding (Source: sources/2026-07-29-atlassian-inside-the-events-rail).

Operational numbers

Metric Value
Events per month 10 billion+
Sustained throughput ~6,000 events/sec
Peak day 600M+ events
Growth rate ~18% MoM (compounding, 2 quarters)
Production regions Multiple
Customer sites served Hundreds of thousands
Matching pod throughput Tens of thousands evaluations/sec (warm cache)
Delivery pod concurrency Hundreds of concurrent outbound HTTP exchanges

Architecture

Atlassian Products → Internal Event Substrate (schema validation, durable persistence)
                                    ↓
                            Events Queue
                                    ↓
                    ┌─── Event Matching (CPU-bound) ───┐
                    │  - Subscription registry lookup   │
                    │  - Negative-lookup cache          │
                    │  - Producer/consumer criteria     │
                    │    evaluation (filter expressions) │
                    └─────────────┬────────────────────┘
                                  ↓
                           Webhooks Queue
                                  ↓
                    ┌─── Event Delivery (network-bound) ─┐
                    │  - Per-tenant rate limiting         │
                    │  - Per-recipient concurrency limit  │
                    │  - Circuit breakers on upstreams    │
                    │  - Exponential backoff + jitter     │
                    │  - Dead-letter queue per pipeline   │
                    └──────────────┬─────────────────────┘
                                   ↓
                    Forge Runtime / Customer URLs / Audit-log endpoints

Systems extracted

Concepts extracted

Patterns extracted

Source

Last updated · 602 distilled / 1,824 read