Skip to content

SYSTEM Cited by 2 sources

Netflix Data Gateway Platform

Netflix Data Gateway is the platform layer underneath Netflix's Data Abstraction Layer (DAL) services. It hosts the Data Abstraction Layer services that sit between microservices and online datastores (Cassandra, EVCache, DynamoDB, RocksDB, etc.) and "enables us to support the broad spectrum of use cases that Netflix demands with minimal developer effort."

Within this wiki, the Data Gateway is a stub page — named as the parent platform in the sources/2024-09-19-netflix-netflixs-key-value-data-abstraction-layer|2024-09-19 KV DAL post but not yet deep-dived. The canonical explanation is Netflix's earlier post, Data Gateway: A Platform for Growing and Protecting the Data Tier, which has not yet been ingested on the wiki — expansion is expected when it is.

Role in the KV DAL story

The 2024-09-19 KV post names the Data Gateway only in passing:

"To overcome these challenges, we developed a holistic approach that builds upon our Data Gateway Platform. This approach led to the creation of several foundational abstraction services, the most mature of which is our Key-Value (KV) Data Abstraction Layer (DAL)."

Two points follow:

  1. KV DAL is not the only abstraction service on the Data Gateway. The post says "several foundational abstraction services" — KV is the mature exemplar, but other DALs (e.g. time-series, graph, etc.) presumably run on the same platform.
  2. Data Gateway provides the shared infrastructure on which DAL services can enforce the cross-cutting properties KV relies on: namespace-routed storage, in-band signaling, consistency-scope / consistency-target config, policy-layer access control.

Mature abstractions named

As of the 2024-11-13 Distributed Counter post, the wiki can name three mature abstractions hosted on the Data Gateway, with direct composition between them:

  1. KV DAL — gRPC point-lookup service with a two-level-map data model over Cassandra + EVCache + DynamoDB + RocksDB.
  2. TimeSeries Abstraction — event store for temporal event data, backed by Cassandra with bucketed partitioning.
  3. Distributed Counter Abstraction — counting service built on top of TimeSeries using it as its event store + EVCache as rollup cache. This composition — one DAL consuming another — is the concrete proof of the platform's "compose multiple abstraction layers using containers deployed on the same host, with each container fetching configuration specific to its scope" property.

Seen in

Last updated · 319 distilled / 1,201 read