Skip to content

SYSTEM Cited by 2 sources

LSVD (Log-Structured Virtual Disk)

LSVD is a log-structured virtual disk architecture in which local NVMe is used as a cache for a durable object- storage backend, rather than as the durable storage tier itself. Writes land in an append-only log in local NVMe and are asynchronously propagated to object storage (S3-shaped); durability is eventually anchored at the object-storage tier (S3-grade 11-nines durability), and read latency is preserved by the NVMe cache for hot blocks.

Role in Fly.io's storage roadmap

In the Making Machines Move post, LSVD is Fly.io's stated medium-term storage direction:

We're a lot more interested in log-structured virtual disks (LSVD). LSVD uses NVMe as a local cache, but durably persists writes in object storage. You get most of the performance benefit of bus-hop disk writes, along with unbounded storage and S3-grade reliability.

Fly launched LSVD experimentally in 2023 as "bottomless S3-backed volumes." The 2024 architectural inflection is Tigris: "Tigris Data launched S3-compatible object storage in every one [of] our regions, so instead of backhauling updates to Northern Virginia, we can keep them local." Regional object storage makes LSVD much more attractive, because the S3 persistence hop stays in-region.

Architectural shift

  • Before: local NVMe is the durable tier. A Fly Volume is anchored to a worker; if the worker dies, the volume is lost unless snapshotted to off-network storage. Migration is expensive (this whole post).
  • With LSVD: local NVMe is a cache; the durable tier is regional object storage. Worker loss = cache invalidation, not data loss. Migration becomes "rehydrate the cache on a different worker" rather than "copy the durable data."

Rewrites concepts/bus-hop-storage-tradeoff: you keep the bus-hop read performance on the hot path while removing the "anchored to a specific worker physical" migration cost.

Seen in

  • sources/2024-07-30-flyio-making-machines-move — Named as Fly's stated medium-term storage direction. "We have more to say about LSVD, and a lot more to say about Tigris."
  • [[sources/2026-01-14-flyio-the-design-implementation-of- sprites]] — referenced as the Fly-Machine-side block-level precursor whose performance ceiling Sprites route around. Ptacek: "We've always wanted to run Fly Machine disks off object storage ([we have an obscure LSVD feature that does this]), but the performance isn't adequate for a hot Postgres node in production." LSVD stays experimental; Sprites take the filesystem-level route via JuiceFS for workloads that don't need OLTP-grade write latency. Establishes LSVD's status: 2-3 years after the 2023 experimental launch and the 2024 "medium-term storage direction" framing, LSVD still hasn't displaced attached- NVMe Fly Volumes on the Fly-Machines side, and Sprites are the product that ships the object-store-rooted-disk idea for a different workload shape.

Caveats / open questions

  • LSVD is "more interested in," not "we're shipping it." The 2023 launch was explicitly experimental; production uptake, performance, and the migration story off local-NVMe-as-durable are not in this post.
  • Write latency characteristics of an NVMe-cache-in-front-of-S3 design under various workloads are not disclosed.
  • Consistency / durability contract at the NVMe-log level vs. the S3-persisted level is not described.
  • systems/fly-volumes — The incumbent primitive LSVD would eventually displace.
  • systems/tigris — The regional S3-compatible object store that makes LSVD much more attractive for Fly.io.
  • systems/aws-s3 — The shape LSVD's persistence tier imitates.
Last updated · 542 distilled / 1,571 read