Skip to content

SYSTEM Cited by 2 sources

Netflix TimeSeries Abstraction

Netflix TimeSeries Abstraction is a distributed service on the Data Gateway platform for storing and querying large volumes of temporal event data with low millisecond latencies. Announced in a pre-2024-11-13 post (Netflix TimeSeries Data Abstraction Layer) and canonicalised on this wiki via its load-bearing role as the event store underneath Netflix's Distributed Counter service. (Source: sources/2024-11-13-netflix-netflixs-distributed-counter-abstraction; named earlier in sources/2024-09-19-netflix-netflixs-key-value-data-abstraction-layer.)

Within this wiki, TimeSeries is currently a first-class system page but most of its internals are disclosed only indirectly through the Counter post. A dedicated deep-dive on TimeSeries would expand this page substantially.

Role on the platform

TimeSeries is the second mature abstraction on the Data Gateway, sibling to KV DAL, and is composed by downstream services as their event store. The Distributed Counter Abstraction uses TimeSeries directly as its backing event store, demonstrating Netflix's "compose multiple abstraction layers using containers deployed on the same host" Data-Gateway property.

Data model + schema shape (as used by Counter)

Each TimeSeries event carries at least:

  • event_time — the logical timestamp of the event; also doubles as part of the idempotency key.
  • event_id — per-event identifier (can encode source provenance).
  • event_item_key — further key scoping within an event.
  • time_bucket, event_bucket — schema-level partitioning columns that break up otherwise-wide Cassandra partitions (see patterns/bucketed-event-time-partitioning).
  • Payload columns for the event contents (e.g. the counter delta).

The composite (event_time, event_id, event_item_key) forms a natural idempotency key: retried and hedged writes with the same token produce the same row.

Key operational properties (disclosed via the Counter post)

  • Backed by Cassandra as the underlying event store. Can be configured to work with other persistent stores — the Counter-Rollup Store reuses TimeSeries abstraction but with different tuning.
  • Wide-partition prevention via bucketing: time_bucket + event_bucket columns prevent high-throughput events for a given identifier from overwhelming a single Cassandra partition. See concepts/wide-partition-problem.
  • Descending time order: TimeSeries returns events in descending event-time order. Counter Abstraction exploits this for reset semantics — the most recent ClearCount event is seen first when scanning the window.
  • acceptLimit bounds event skew: incoming events with timestamps beyond this limit (example config: 5 s) are rejected. This doubles as the safety lag under which downstream consumers can treat a time window as an immutable aggregation window.
  • Retention policies (lifecycle config): events close after close_after seconds and are deleted after delete_after seconds. Example config: 6 days close, 7 days delete. Events aggregated into a downstream rollup store don't need to live in TimeSeries past the audit window.
  • Range-scan optimisation for low millisecond latency on aggregation workloads — Counter-Rollup issues parallel range queries across time partitions per batch.

Seen in

Last updated · 319 distilled / 1,201 read