Skip to content

PATTERN Cited by 1 source

S3 as Single Source of Truth

Definition

Rather than querying multiple services to determine backup existence and completeness, read metadata directly from the backup storage layer (S3) itself — collapsing a fragile multi-service dependency chain into a single, authoritative source of truth.

Canonical example

Netflix's Cassandra data-movement architecture (Source: sources/2026-06-19-netflix-the-evolution-of-cassandra-data-movement-at-netflix):

  • Regularly scheduled backups are performed on Cassandra nodes via a sidecar process that uploads SSTables and metadata files directly to S3.
  • The new engine reads backup existence and structure by referencing the S3-based metadata, bypassing the multiple services Casspactor relied on.
  • Result: eliminated silent metadata divergence and fragile coordination requirements.

Key insight

"The answer to 'which backup exists and is it complete?' already lived in the backup storage layer (Amazon S3) itself."

When to apply

  • When backup/artifact metadata is derivable from the artifacts themselves.
  • When a composite metadata view has become a reliability liability.
  • When the storage layer (S3, GCS, etc.) already provides strong consistency and durability guarantees.

Seen in

Last updated · 546 distilled / 1,578 read