Skip to content

SYSTEM Cited by 1 source

Lyft dsfeatures

dsfeatures (short for "data science features") is the online serving layer of the Lyft Feature Store. It is an optimized wrapper over three AWS data stores, exposing one CRUD API via SDKs so every consumer — internal Airflow DAGs, customer services, streaming feature writers — hits the same surface regardless of ingestion lane.

Backing stores

  • DynamoDB — persistent source for all feature data. Primary key is a composite of metadata fields. A Global Secondary Index (GSI) is maintained specifically for GDPR-deletion efficiency so bulk-delete-by-user is bounded-cost.
  • ValKey write-through LRU cache — sits on top of DynamoDB for ultra-low-latency retrievals of the most frequently-accessed (meta)data. "Generous TTL." The write-through shape keeps DynamoDB authoritative: a write lands in DynamoDB and the cache in order; a cache miss always has a correct DynamoDB read behind it.
  • OpenSearch — serves embedding features only; other feature types don't live here. OpenSearch is chosen because kNN / vector similarity retrieval is native to it and foreign to DynamoDB.

API surface

Two SDKs:

  • go-lyft-features (Go)
  • lyft-dsp-features (Python)

Both expose full Create / Read / Update / Delete. The most common retrieval methods are Get / BatchGet. Write access is used by:

  • Internal Airflow DAGs — the batch lane's online-write step.
  • spfeaturesingest — the central Flink ingest app calls dsfeatures WRITE API on behalf of customer streaming applications.
  • Customer services — ad-hoc management of real-time features (the direct-CRUD lane).

Design properties

  • Strongly consistent reads across ingestion lanes. "Regardless of the ingestion method (batch, streaming, or on-demand), the Feature Store maintains uniform metadata and strongly consistent reads."
  • Feature-type-specific routing behind a uniform interface. Callers don't pick a store — dsfeatures routes to DynamoDB / ValKey / OpenSearch based on feature type. This is the wrapper-over-heterogeneous-stores pattern's canonical win.
  • GSI-for-GDPR is an explicit schema decision. The post names GDPR-deletion efficiency as a motivating constraint for the secondary index — a concrete operator-facing ergonomics bake-in.

Caveats

  • Write-through failure mode not disclosed. What happens if DynamoDB succeeds and ValKey invalidation/update fails? The post doesn't say; in practice this is the unglamorous hard part of every write-through cache.
  • No published latency numbers. "Ultra-low-latency" is stated but p50/p99/p999/QPS/cache-hit-rate are not disclosed.

Seen in

Last updated · 319 distilled / 1,201 read