Skip to content

SYSTEM Cited by 1 source

LSVD (Log-Structured Virtual Disk)

LSVD is a log-structured virtual disk architecture in which local NVMe is used as a cache for a durable object- storage backend, rather than as the durable storage tier itself. Writes land in an append-only log in local NVMe and are asynchronously propagated to object storage (S3-shaped); durability is eventually anchored at the object-storage tier (S3-grade 11-nines durability), and read latency is preserved by the NVMe cache for hot blocks.

Role in Fly.io's storage roadmap

In the Making Machines Move post, LSVD is Fly.io's stated medium-term storage direction:

We're a lot more interested in log-structured virtual disks (LSVD). LSVD uses NVMe as a local cache, but durably persists writes in object storage. You get most of the performance benefit of bus-hop disk writes, along with unbounded storage and S3-grade reliability.

Fly launched LSVD experimentally in 2023 as "bottomless S3-backed volumes." The 2024 architectural inflection is Tigris: "Tigris Data launched S3-compatible object storage in every one [of] our regions, so instead of backhauling updates to Northern Virginia, we can keep them local." Regional object storage makes LSVD much more attractive, because the S3 persistence hop stays in-region.

Architectural shift

  • Before: local NVMe is the durable tier. A Fly Volume is anchored to a worker; if the worker dies, the volume is lost unless snapshotted to off-network storage. Migration is expensive (this whole post).
  • With LSVD: local NVMe is a cache; the durable tier is regional object storage. Worker loss = cache invalidation, not data loss. Migration becomes "rehydrate the cache on a different worker" rather than "copy the durable data."

Rewrites concepts/bus-hop-storage-tradeoff: you keep the bus-hop read performance on the hot path while removing the "anchored to a specific worker physical" migration cost.

Seen in

Caveats / open questions

  • LSVD is "more interested in," not "we're shipping it." The 2023 launch was explicitly experimental; production uptake, performance, and the migration story off local-NVMe-as-durable are not in this post.
  • Write latency characteristics of an NVMe-cache-in-front-of-S3 design under various workloads are not disclosed.
  • Consistency / durability contract at the NVMe-log level vs. the S3-persisted level is not described.
  • systems/fly-volumes — The incumbent primitive LSVD would eventually displace.
  • systems/tigris — The regional S3-compatible object store that makes LSVD much more attractive for Fly.io.
  • systems/aws-s3 — The shape LSVD's persistence tier imitates.
Last updated · 200 distilled / 1,178 read