Skip to content

CONCEPT Cited by 1 source

Demand-driven replication

Demand-driven replication is the policy of materialising a replica of an object in a region only when that region actually reads the object, rather than proactively mirroring every write to every region. It sits between two classical extremes:

  • Eager full replication — every write is synchronously (or quickly asynchronously) copied to every replica region. Low read latency everywhere, but write amplification and storage cost scale with the number of regions × the write rate, regardless of whether any region actually wants the data.
  • Single-region origin + CDN cache — one authoritative region; reads elsewhere go through a caching CDN that pulls on miss. Read latency is high on first access per region, cache coherence / invalidation is a separate concern, and there is no strong-consistency story across regions.

Demand-driven replication splits the difference: writes land in one (or a few) regions; the system learns which regions want an object by observing read traffic, and materialises replicas in those regions. Subsequent reads in the same region are served locally with data-plane latency characteristics.

Canonical example — Tigris

Tigris's Fly.io-announced architecture:

  • Objects <~128 KB: proactively global — small-enough objects are pushed to every region at write time. The bytes are cheap to ship; the latency benefit of having them everywhere is high.
  • Objects ≥ ~128 KB: demand-driven — larger objects stay in origin regions until another region reads them, at which point they propagate to that region via the QuiCK-style queue. Tigris explicitly describes this as "to distribute object data to multiple replicas, [and] to regions where the data is in demand".
  • Regional metadata stays authoritative in FoundationDB — lookups can always tell a region whether it has a local copy or needs to pull.

(Source: sources/2024-02-15-flyio-globally-distributed-object-storage-with-tigris)

When it makes sense

  • Skewed / local demand per object. The hamdog-photo-in-Australia workload: the readers of any individual object are mostly geo-correlated with each other, even in a globally-distributed system. A per-object replica in the correlated region is strictly better than a central origin or full-fanout.
  • Large objects with low cross-region access rate. Shipping the bytes everywhere on write is wasteful; shipping them on first remote access is roughly free on the long run (amortised across many subsequent reads in that region).
  • Asymmetric write / read patterns. Writes are rare or concentrated; reads are distributed. The pay-per-demand-region trade beats either extreme.
  • Egress cost sensitivity. Eager full replication pays cross-region egress for data that may never be read in those regions. Demand-driven replication pays egress exactly when a region asks — aligning cost with value.

When it doesn't

  • Strict read-latency SLO on first access in a region. If every read must be sub-10 ms no matter where, demand-driven degrades into a CDN-miss shape on first touch. The threshold- based policy (instant-global for small objects, demand-driven for large) is the common compromise.
  • Cross-region strong consistency on writes. Demand-driven is a byte-plane replication policy; it implicitly assumes writes can be routed to a specific region and that clients tolerate some asynchrony on the propagation path. Systems that need strict multi-region write consistency usually can't use a pure demand-driven model — they need synchronous replication to a quorum.

Relationship to adjacent concepts

  • concepts/geographic-sharding — demand-driven replication usually rides on top of a geographic partitioning of metadata + primary placement. The shard tells you where the authoritative copy lives; demand-driven tells you where secondary copies materialise in response to traffic.
  • patterns/caching-proxy-tier / CDN caching — both materialise local copies on demand. Demand-driven replication differs by treating the local copy as a first-class replica in the coherence protocol, not a cache entry with a TTL and a miss-then-fetch semantics. Tigris's framing: "a toolset that you can use to build arbitrary CDNs, with consistency guarantees, instant purge and relay regions" — i.e. this is replication-with-coherence, not caching-with-invalidation.
  • concepts/immutable-object-storage — immutability drastically simplifies demand-driven replication: a replica is either the full (identical) object or absent; there is no partial-state race to resolve.

Seen in

Last updated · 200 distilled / 1,178 read