--- title: Inside the Events Rail: How Atlassian Delivers 10 Billion Webhooks a Month source: Atlassian Engineering source_slug: atlassian url: https://www.atlassian.com/blog/how-we-build/inside-atlassians-webhook-delivery-platform published: 2026-07-29 fetched: 2026-07-29T14:26:09+00:00 ingested: true --- Atlassian products generate a lot of events. _Your_ Jira issue gets created, _your_ Bitbucket pull request is merged, _your_ Confluence page is published, a workflow transition fires, an admin action lands in the audit log. **Forge (Atlassian’s extensibility platform for building apps on top of products like Jira, Confluence, and Bitbucket) is the largest consumer of those events** , and getting them delivered to Forge apps (and to a long tail of repository hooks, audit-log endpoints, and workflow post functions) reliably, securely, and at Atlassian scale is the job of the webhooks-processor service. We think of it as Atlassian’s _events rail_ , the multi-tenant infrastructure that evaluates, matches, and delivers events from Atlassian products to the apps and systems that subscribe to them. The rail is built so that adding a new event source or destination doesn’t require rebuilding the platform underneath it. Today, the rail does a lot more than it used to. ![](https://atlassianblog.wpengine.com/wp-content/uploads/2026/07/dashboard-events.png) * Handles **over 10 billion events per month** , with sustained throughput of around 6,000 events per second, with peak days above 600 million events * Runs across **multiple production regions** , serving hundreds of thousands of customer sites * Delivers to a large and growing population of Forge apps alongside customer-configured webhook subscriptions across many products * Carries the full range of Atlassian event traffic, including product events, lifecycle events, async app events, repository events, workflow events, and audit-log events * Runs with a **high-availability reliability target** for the delivery surfaces that depend on it. * Continues to grow at **~18% month over month** , compounding This post is a tour of how it’s built, covering the architecture, the hard problems that come with running this kind of system at Atlassian’s scale, and the design decisions that keep it predictable. * * * ## Why webhook delivery at Atlassian scale is a hard problem “Webhook delivery” sounds straightforward (receive an event, look up subscribers, POST to a URL), but three properties make it a hard problem at our scale: 1. **Subscriptions are highly selective.** Apps don’t subscribe to “all events from Atlassian.” They subscribe to specific event types, in specific tenants, often with additional filter expressions on event attributes. A Forge app subscribes to `issue_updated` events. On top of that, it declares filters over the event payload – for instance, only deliver when `project.key` is `FOO` and `priority` is `Critical`. That predicate rules out the vast majority of issue-updated traffic in that tenant. Evaluation runs on every event; delivery runs only for the matches. In steady state, a large fraction of inbound events are precisely scoped enough that no subscription matches. That’s selective pub-sub working as intended. The implication is operational. **Subscription evaluation, not network I/O, is the cost centre we have to optimize first** , and the no-match path has to be the cheapest path of all. 2. **Multi-tenant fairness is not optional.** A bulk operation in one tenant (a 10,000-issue CSV import, a security policy sweep touching every project) can produce more event volume in a minute than that tenant produces in a normal week. Without explicit fairness controls, that volume would compete with everyone else for delivery capacity. There is no version of this service where “every tenant gets predictable service” happens by accident. 3. **Design Philosophy.** Webhooks-processor is the rail underneath many event surfaces, each with its own developer-facing guarantees published in the Forge documentation. The platform’s job is to make those per-surface guarantees credible at scale, regardless of which surface a particular event came from. The internal design principle is simple: a silently dropped event is a much worse failure than a slightly delayed one. Almost every choice below follows from that. These three constraints shape almost every design choice that follows. * * * ## Architecture overview At a high level, webhooks-processor sits between three worlds: * **Producers** : Atlassian products (Jira, Confluence, Bitbucket) publishing events through an internal publish/subscribe substrate, with a few REST-triggered flows on the side. * **A two-stage processing pipeline**. _Event Matching_ reads from an events queue and writes matched events onto a webhooks queue; _Event Delivery_ drains the webhooks queue and dispatches. * **Consumers** : Forge apps via the Forge runtime, repository-hook recipients at customer-controlled URLs, and audit-log endpoints at customer-supplied destinations. ![](https://atlassianblog.wpengine.com/wp-content/uploads/2026/07/d1.png) ### Producers emit events on shared rails Atlassian products don’t talk to webhooks-processor directly. They publish to an internal event substrate which validates each event against a registered schema and durably persists it. We declare subscriptions against that substrate, and its distribution layer fans events out to our Events Queue. Across all sources the shape of the work is the same. Typed events with strong schemas and clean attribution back to the originating site, app, and tenant. ### Event Matching decides who cares When an event arrives, the matching component asks two questions in order: 1. **Are there any subscriptions for this event?** A lookup against the subscription registries. Forge subscriptions live in a tenant context service; product-configured subscriptions (such as repository hooks) live with the originating product. 2. **For each candidate subscription, do the producer and consumer criteria both match?** Producer criteria express constraints from the event source, for example, “this event is sensitive; only deliver to apps with the right access narrowing.” Consumer criteria express constraints from the subscriber, filter expressions on event attributes that let apps say “only notify me about high-priority bugs in project X.” Matched events become messages on the Webhooks Queue. Unmatched events are not delivered, fast, on purpose, and with metrics so we can see how much of the inbound stream they represent. ### Event Delivery is where the network meets reality The delivery component pulls messages off the queue and dispatches them. For Forge apps, that means routing into the Forge runtime. For repository-hook and audit-log recipients, that means an outbound HTTPS POST to a customer-controlled URL. Between “I have a delivery to make” and “the recipient acknowledged” sit several reliability primitives: rate limiting, retries with exponential backoff, and a dead-letter queue for messages that can’t be delivered after retries. These aren’t optional features bolted on top. They _are_ the delivery component. ### One core, many event types A service that handles Forge product, lifecycle, and async app events alongside repository events, audit-log events, and workflow events could very easily become a tangle of `if (eventType == X)` branches. Instead, the service is built around a **per-surface configuration model**. The core processing pipeline exposes well-defined interfaces, for matching, for delivery, for retry policy. And each event type supplies its own implementation. Each event surface declares its own destination lookup, routing, retention semantics, filtering, and failure behaviour against the same core interfaces. The core knows nothing about any of those specifics. Adding a new event surface is a self-contained change, not an invasive one. That’s what lets the rail keep growing without re-litigating the platform every time a new product or surface needs webhook delivery. * * * ## Two stages, two queues The processing pipeline is deliberately split into two stages with queues between them. Event Matching consumes from an **Events Queue** fed by producers and writes matched messages onto a **Webhooks Queue** ; Event Delivery consumes from the Webhooks Queue and dispatches. That shape does a lot of structural work, and it’s worth examining why. ### The two stages have very different scaling characteristics Matching is **CPU-bound**. For each inbound event it runs a registry lookup, an expression evaluation against subscriber criteria (path traversal, equality, and boolean composition over event attributes), and producer-side filtering for sensitive content. A matching pod’s working set is dominated by the subscription caches; what it wants is fast access to those caches and enough cores to evaluate criteria at line rate. A single matching pod sustains tens of thousands of evaluations per second when caches are warm. Delivery is **network-bound**. It opens connections to external endpoints that vary wildly in latency, some Forge apps respond in tens of milliseconds, some customer-controlled URLs take seconds. A delivery pod’s working set is dominated by **in-flight requests** and the file-descriptor and concurrency budget that bounds them. What it wants is generous concurrency to absorb tail latency without head-of-line blocking, and a small CPU footprint per in-flight request. A typical delivery pod sustains hundreds of concurrent outbound HTTP exchanges. Combining matching and delivery in one process would force every worker to carry both sets of resources whether it needed them or not. Splitting them lets each component scale on the queue that feeds it, on the dimension that actually matters for its work. ### The queues are the contract Putting queues between stages doesn’t just decouple them operationally — it gives us four properties that matter: 1. **Independent scaling.** Each stage runs as a separate worker group that autoscales based on the depth of the queue feeding it. 2. **Flow control as a first-class signal.** A pod holds a permit for every in-flight delivery, releasing it only on completion – so a run of slow deliveries naturally throttles how fast that pod pulls more work off its queue. Autoscaling handles the rest, adding capacity as queue depth grows. 3. **Contained blast radius.** A slow or unresponsive downstream recipient stays a delivery-side concern. Matching keeps producing decisions and the Webhooks Queue absorbs them, so the impact is bounded to the slice of traffic destined for the affected recipient – each recipient has its own concurrency limit. 4. **A clearer mental model.** New code goes into matching if it’s about _“who should get this event”_ and into delivery if it’s about _“how do we get it there”_ , a clean split that scales as the team and the surface area grow. We didn’t always have this shape. An earlier generation of the service ran matching and delivery in a single process, simpler to reason about until a single slow recipient could stall the matching loop behind it. The two-stage split was a direct response to that failure mode. ![](https://atlassianblog.wpengine.com/wp-content/uploads/2026/07/d2.png) ## Multi-tenant fairness This is the part of the system where most of the design work lives, and where webhooks-processor diverges most from a textbook event bus. Multi-tenant fairness here is **layered, not centralized** : four distinct mechanisms, each catching a kind of pressure the one before it doesn’t – a negative-lookup cache, queue isolation between pipelines, per-tenant rate limiting at the leaf, and per-recipient concurrency limiting for slow responders. ![](https://atlassianblog.wpengine.com/wp-content/uploads/2026/07/d3-new-scaled.png) **What each layer catches that the layer above didn’t:** * **The negative-lookup cache** catches _evaluation cost_ : events that would otherwise waste a registry lookup and a criteria evaluation only to find no subscriber. * **Queue isolation** catches _head-of-line blocking_ : one workload’s burst that would otherwise queue behind itself and delay unrelated traffic sharing the same pipe. * **Per-tenant rate limiting** catches capacity exhaustion: a single noisy app or tenant that, even after queue separation, would consume more outbound capacity than the platform can sustainably give it. * **Per-recipient concurrency limiting** catches recipient slowness: a recipient that isn’t receiving requests at a high rate, just taking too long to answer each one, never trips a rate limit, and only shows up as pressure here. ### Making subscription evaluation cheap when most events have no subscriber Subscriptions are highly selective. An app might subscribe only to “issue updated” events, only in a specific tenant, only when a specific field changed. That selectivity is the _feature_ , apps and customers want precise targeting. But it has a runtime implication. **Evaluation runs on every event, delivery runs only for the matches.** The evaluation path is hot, and making it efficient is where a lot of the engineering attention goes. Subscription registries answer two kinds of question, _are there any subscriptions for this event?_ and, if yes, _which ones?_ For the high-volume event types where most events have no subscriber, the first question is the one we ask at very high volume, and the answer is overwhelmingly “no subscriptions.” Asking the registry every time is wasted work for both sides. And at this volume, wasted work shows up as real cost on real graphs. The **negative-lookup cache** stores those “no subscriptions” answers behind a short TTL, long enough to absorb the burst behaviour of high-volume event types, short enough that a newly-registered subscription takes effect quickly. It is applied selectively to the event types where the no-match rate is consistently high, so freshness for general-purpose event types is unaffected. The TTL is the lever we tune. Shorter TTL trades cache hit rate for subscription freshness; longer TTL does the reverse. The pattern is worth naming because it shows up in many selective pub-sub designs as **Negative-result caching for high-cardinality, miss-heavy lookups.** Most cache discussions focus on hit-side payoff; in workloads like ours, the _miss side_ is the win, because the cost of the underlying lookup is paid on every event and most of those lookups return nothing. The result is a substantial reduction in subscription-registry calls for the targeted event types, with no change to the contract subscribers see. ### Queue isolation between unrelated workloads A single delivery queue means one workload’s traffic waits behind another’s, classic head-of-line blocking, and a problem we deliberately chose to engineer around rather than tolerate. The approach is **queue isolation**. Each pipeline – product events, lifecycle events, Bitbucket repository hooks, audit-log delivery, and so on – gets its own physically separate queue and its own independently deployed worker fleet. A burst in one pipeline can only consume its own fleet’s capacity; it has no path to slow down another pipeline’s pods, because they’re different deployments entirely. What separates pipelines isn’t a customer-visible “fast vs slow” label. It’s the operational characteristics each event surface needs. Different surfaces have different timeout budgets, retry policies, and failure-recovery shapes; keeping their queues isolated means one surface’s worst day stays inside its own boundary. The judgment that matters is how finely to partition: too coarse and unrelated workloads still share a fleet; too fine and you pay in operational overhead for fleets that rarely see contention. ### Per-tenant rate limiting at the leaf The third layer is a distributed rate limiter on outbound delivery, applied per-app-and-environment and per-tenant, with a separate bucket at each scope – enforced by a shared, cluster-wide service, so the limits hold consistently even though delivery runs across many pods. Its job here is **system protection** more than per-tenant policy. A webhook recipient that suddenly accepts deliveries far slower than normal, an unexpected spike in subscriptions for a high-volume event type, or an app subscribing more broadly than its capacity to receive — without rate limits, any of these can pull more outbound capacity than the platform has to give and degrade delivery for everyone else sharing the same path. The rate limiter is the backstop that keeps the platform’s overall capacity healthy regardless of how individual apps or tenants behave. Per-app-and-environment and per-tenant both matter. The per-app-and-environment cap keeps any single subscription from consuming more than its share; the per-tenant cap keeps any single tenant’s collective subscriptions from doing the same. Both buckets must allow capacity for the delivery to proceed. When capacity is exhausted, the delivery is deferred, not dropped, and re-evaluated as the bucket refills, with the deferral policy chosen per pipeline. ### Per-recipient concurrency limiting for slow responders Per-tenant rate limiting catches volume – too many requests. It doesn’t catch latency: a recipient receiving requests at a completely normal rate, but taking ten seconds to answer each one, never trips a rate limit. Left unchecked, that recipient can quietly tie up delivery capacity the rest of the platform needs, one slow response at a time. The fix is a concurrency limit, not a rate limit: capping how many deliveries to a given recipient can be in flight at once, tracked in a shared, cluster-wide counter so the limit holds no matter which pod picks up the next message for that recipient. When a recipient is healthy, deliveries flow through untouched. When a recipient slows down, its in-flight deliveries pile up against its own cap, and further deliveries to it are deferred rather than dispatched – exactly the mechanism **contained blast radius** depends on earlier in this post. The limit is scoped per recipient, so it’s applied selectively wherever a concurrency budget is configured, not globally, and each reservation carries a short expiry, so a pod crash or a dropped release doesn’t leave a recipient’s capacity permanently stuck. Together, queue isolation, per-tenant rate limiting, and per-recipient concurrency limiting are what let the rail absorb large, uncoordinated load from many tenants – and outright unresponsive recipients – without the system itself becoming the bottleneck. * * * ## How delivery stays reliable A few smaller pieces that don’t need their own deep dive but matter to the whole picture, the everyday discipline that keeps deliveries landing under load, under failure, and under change: **Stable identifiers and idempotency-friendly delivery.** Every webhook delivery carries a stable identifier that remains constant across retries, so recipients can deduplicate. The identifier is scoped per event-instance, not per attempt, so a recipient that sees the same key twice can confidently treat the second as a duplicate. Retries follow **exponential backoff with jitter** , capped at a small number of attempts, after which the message moves to a per-pipeline **dead-letter queue** for inspection and replay. The exact retry budget, attempt count, and timeout vary per event surface. For the surfaces Forge exposes to developers, semantics are described in the Forge developer docs. The platform’s job is to honour the policy the surface declares. **Back-pressure on unhealthy paths.** When a downstream (an internal upstream we depend on, or a recipient endpoint that has started struggling) shows sustained errors or latency, the system reduces the rate at which it sends work in that direction. When the signals recover, it gradually opens back up. The asymmetry is deliberate. Pulling back fast and recovering slow is what keeps a downstream’s bad minute from amplifying into a delivery storm. This is the same shape of back-pressure that TCP congestion control uses, applied at the application layer. **Graceful degradation when upstream dependencies stall.** Delivery itself depends on a few internal upstreams: extension lookup, destination resolution, the rate limiter. We wrap each call with a **circuit breaker** so that when an upstream gets slow or unhealthy, calls fail fast instead of locking up worker threads waiting on it. A single dependency hiccup degrades into the breaker’s scoped failure mode rather than cascading into a system-wide delivery stall. Recipients see a delayed delivery from the affected pipeline, not a stalled rail. **Pipelines stay isolated.** Different event surfaces get different pipelines, so the operational properties one surface needs (retry budget, failure handling, scheduling policy) never leak into another. Mixing surfaces with different semantics into a single queue is one of the cleanest ways to create silent failures, so we don’t. **Observability is non-negotiable.** Every delivery emits structured logs (capped at a per-event size limit so logging never becomes the bottleneck), metrics tagged by deployment, region, event type, recipient class, and outcome, plus a trace tag that lets a single delivery be followed end-to-end across both stages. SLI dashboards and SLOs are the daily working surface for the team. * * * ## Sizing for what’s next Volume on the rail is currently growing **~18% per month, compounding** — a trend that has held across every production region for the last two quarters. At that rate, today’s peak is next year’s baseline. The reasons we’ve designed for layered fairness, predictable backpressure, and protection from upstream stalls aren’t just about today’s traffic. Two structural shifts are increasing load on the rail in ways that change the planning math. And they’re the reason this work is more relevant now, not less. **AI agents working inside Atlassian products.** As agents become first-class users of Jira, Confluence, and Bitbucket, creating issues, adding comments, opening pull requests, transitioning workflows, every one of those actions is an event the rail has to evaluate and, where a subscription matches, deliver. Agent traffic doesn’t behave like human traffic. It arrives in bursts, it’s machine-paced, and it can be an order of magnitude denser than the equivalent human activity in the same tenant. The negative-lookup cache, the queue isolation between pipelines, the per-tenant rate limits and the per-recipient concurrency limits are precisely the layers that absorb that shape of load without making the rail visible to anyone else on it. **Customers migrating from Data Center to Cloud.** Many large Atlassian customers are in the middle of moving from self-hosted Data Center to Cloud, and as they bring their full event traffic onto the rail, the per-tenant volume on a single migration day is a step-change above their steady state. The discipline of treating multi-tenant fairness as a designed-in property, rather than something the system has to grow into after each migration, is what lets the platform absorb migration after migration without each one becoming an incident. The architecture in this post wasn’t built for a static world. It was built so that the next order-of-magnitude in volume, whatever’s driving it, doesn’t require rebuilding the platform underneath the apps and customers depending on it. * * * ## Conclusion Webhook delivery at Atlassian’s scale rewards a particular kind of design discipline. If we had to compress the lessons of this platform into reusable maxims, they would be these three: 1. **In selective pub-sub, optimise the no-match path before the match path.** It runs more often, and its cost dominates total throughput. Negative-lookup caching is the cheapest way to honour this maxim without giving up freshness guarantees subscribers actually need. 2. **Flow control is a first-class signal, not a side effect.** A queue between two stages turns “delivery is slow” into a measurable, actionable input that drives autoscaling and absorbs downstream pressure before it becomes visible. Without that queue, slowness is invisible until it’s painful. 3. **Multi-tenant fairness is layered, not centralized.** A negative-lookup cache that protects evaluation, queue isolation that prevents head-of-line blocking, per-tenant rate limits that protect overall system capacity, and per-recipient concurrency limits that catch slow-but-not-noisy recipients – each catch a different kind of pressure. No single layer is sufficient; together they are. Forge webhook delivery shaped this service. The need to deliver product, lifecycle, and async app events to a large and growing population of Forge apps, across hundreds of thousands of customer sites, predictably, within the per-surface objectives the Forge docs publish, is what drove the architectural choices above. The two-stage split, the layered fairness model, the per-surface configuration, the explicit per-pipeline contracts. The same rail also carries Bitbucket repository hooks, Jira workflow post functions, and Jira Cloud audit-log delivery, every consumer running on the same predictability, the same fairness, the same observability. That’s the payoff of building one well-engineered events rail under every event type that leaves Atlassian. New surfaces don’t ship with a minimum-viable webhook. They inherit the same delivery discipline the rail was built for, from day one. Fan-out scheduling, multi-tenant backpressure, subscription evaluation at scale, the engineering of extensibility platforms. If you’re working on any of these, we’d love to hear what you’ve learned.