CONCEPT Cited by 1 source
Metric temporality (delta vs cumulative)¶
Temporality is how a metric data point expresses change over time:
- Cumulative: each point is the running total since the counter
was created (Prometheus default). Deltas are derived at query time
with
rate()/increase(). - Delta: each point is the change during the last export interval (StatsD-style). Deltas are already baked in.
OTel supports both and lets the SDK choose via
AggregationTemporalitySelector (e.g.
AggregationTemporalitySelector.deltaPreferred()).
(Source: sources/2026-04-16-airbnb-statsd-to-otel-metrics-pipeline)
Why the choice matters at scale¶
Cumulative requires the producer to retain full state for every active metric-label combination between exports — it has to know the total to report. On services emitting 10K+ samples/sec per instance, Airbnb observed:
- memory pressure
- increased GC activity
- heap growth
Delta temporality cuts that: each export interval is self-contained, so the SDK doesn't have to hold onto per-series state across exports. Airbnb switched only its highest-volume emitters to delta; the rest of the fleet stayed on cumulative defaults.
Failure-visibility trade-off¶
| Mode | Export failure shows as |
|---|---|
| Cumulative | a jump between two data points (next successful export carries the skipped work) |
| Delta | a gap in the data (the missed delta is simply lost) |
Airbnb accepted gaps in exchange for memory stability on its top emitters. Teams writing alerts over those metrics need to be aware — "no data" alerting semantics differ.
Interaction with rate() and zero injection¶
Even with cumulative temporality, sparse counters can silently
undercount because the first increment happens at creation time and
rate() needs two samples to derive a delta. Airbnb fixes this in the
aggregation tier with patterns/zero-injection-counter. The same
root problem (missing zero baseline) applies here: temporality doesn't
save you from needing a well-defined starting point.
Rules of thumb¶
- Cumulative by default — simpler mental model; matches Prometheus semantics.
- Delta for extreme-cardinality or ultra-high-rate emitters — specifically when per-series retention in the SDK becomes the bottleneck.
- If you pick delta, design alerts to tolerate gaps and instrument export success/failure explicitly.
Seen in¶
- sources/2026-04-16-airbnb-statsd-to-otel-metrics-pipeline — Airbnb moved select high-volume services to delta temporality to fix memory regressions after OTLP dual-write adoption.