Skip to content

SYSTEM Cited by 1 source

Atlassian Events Rail

Atlassian Events Rail (internally: webhooks-processor) is Atlassian's multi-tenant event evaluation, matching, and delivery platform. It sits between Atlassian products (Jira, Confluence, Bitbucket) and event consumers (Forge apps, customer webhook endpoints, audit-log destinations), handling 10 billion+ events/month at ~6,000 events/sec sustained with peaks above 600M events/day.

Architecture

The rail is a two-stage pipeline with queues between stages:

  1. Event Matching (CPU-bound) — reads from the Events Queue, runs subscription registry lookups, evaluates filter expressions (path traversal, equality, boolean composition over event attributes), applies producer-side criteria. Writes matched events to the Webhooks Queue. A single pod sustains tens of thousands of evaluations/sec when caches are warm.

  2. Event Delivery (network-bound) — drains the Webhooks Queue and dispatches via the Forge runtime or outbound HTTPS POST to customer URLs. A single pod sustains hundreds of concurrent outbound HTTP exchanges.

Stages scale independently based on queue depth. The queue is the contract — it decouples scaling, provides flow control as a signal, and contains blast radius from slow recipients.

Multi-tenant fairness (layered)

Four mechanisms, each catching pressure the layer above doesn't:

Layer What it catches Mechanism
1. Negative-lookup cache Evaluation cost (miss-heavy lookups) Short-TTL cache of "no subscriptions" results
2. Queue isolation Head-of-line blocking between surfaces Per-pipeline separate queues + worker fleets
3. Per-tenant rate limiting Capacity exhaustion from noisy tenants Distributed rate limiter (per-app-env + per-tenant buckets)
4. Per-recipient concurrency limiting Slow-but-not-noisy recipients Cap on in-flight deliveries per recipient (cluster-wide)

Key design decisions

  • Per-surface configuration model: each event surface (Forge product events, Bitbucket hooks, audit logs, workflow events) declares its own destination lookup, routing, retry policy, and failure behaviour against shared core interfaces. Adding a new surface is self-contained.
  • Backpressure follows TCP congestion-control shape: pull back fast on errors, recover slowly. Prevents delivery storms.
  • Circuit breakers on every upstream dependency (extension lookup, destination resolution, rate-limiter service).
  • Idempotent delivery: stable per-event-instance identifier across retries; exponential backoff with jitter; per-pipeline dead-letter queue.

Growth

~18% month-over-month compounding growth, driven by AI agents (machine-paced, bursty) and Data Center → Cloud migrations (step-change volume on migration days).

Seen in

Last updated · 602 distilled / 1,824 read