Skip to content

CONCEPT Cited by 1 source

Immutable aggregation window

Definition

An immutable aggregation window is a time range closed for further writes, inside which an event-log aggregation can run without coordination and still converge to the same value from every observer.

Given an event store that rejects incoming events with timestamps beyond a safety limit (an acceptLimit), any time range that ends before now() − acceptLimit is by construction no longer receiving new events. A rollup / reducer / aggregation pipeline can read that range, compute an answer, and write it to a checkpoint store — and because the input set is frozen, every concurrent aggregator produces the same answer.

This is the load-bearing trick underneath Netflix's Distributed Counter Abstraction Eventually Consistent mode. (Source: sources/2024-11-13-netflix-netflixs-distributed-counter-abstraction)

Why it matters

Traditional concurrent aggregation of a growing event log needs distributed locking or linearizable checkpoints to avoid overwriting each other with stale values. Locks hurt availability + tail latency; linearizable checkpoints add coordination overhead.

The immutable-window discipline removes both:

  • Safe over-writes: two Rollup threads reading the same window at slightly different now()s both see the same events, compute the same count, and race harmlessly on the checkpoint write. The later write wins, both writes are correct.
  • No distributed locking: Netflix's Rollup tier can horizontally scale + deploy concurrently without synchronisation — "Although the concept of now() may differ between threads, causing rollup values to sometimes fluctuate, the counts will eventually converge to an accurate value within each immutable aggregation window."
  • Deployment safety: during graceful shutdown + rolling deploys, old + new instances may both issue rollups for the same counter within an overlap period — safe by construction.

Mechanics

Three parameters tune the shape:

  1. acceptLimit — how far into the past an incoming event's timestamp may be. Events beyond this limit are rejected. Example (Netflix Counter): 5 s.
  2. Safety margin — aggregation is performed over events older than now() − acceptLimit − margin to accommodate clock skew across aggregator hosts.
  3. lastRollupTs checkpoint — the most recent time up to which aggregation has completed. The next rollup reads (lastRollupTs, now() − acceptLimit).

Counter Abstraction instance

In the Netflix Counter pipeline:

  • TimeSeries acceptLimit defaults to 5 s.
  • Rollup consumer reads events in (lastRollupTs, now() − acceptLimit − skew_margin) per batch.
  • Aggregate and write (lastRollupCount, lastRollupTs') to the Rollup Store; the lastRollupTs' is the upper bound of the aggregation range.
  • Cached (lastRollupCount, lastRollupTs) in EVCache for fast GetCount.

Cousin concepts

  • Watermarks in stream processors (Flink, Beam) carry the same idea: a watermark t is a promise no event with timestamp < t will arrive.
  • Materialized view refresh windows in time-series databases (e.g. ClickHouse incremental aggregate) need the same immutability property to avoid re-reading + re-publishing partial results.

Trade-off

Inherent staleness: observed counts lag real time by at least the acceptLimit + margin. For Netflix that's on the order of seconds, deliberate. Use cases requiring sub-second accuracy must layer a real-time delta on top (Netflix's experimental Accurate mode computes lastRollupCount + delta(lastRollupTs, now()) in the read path).

Seen in

Last updated · 319 distilled / 1,201 read