CONCEPT Cited by 1 source
Bus-hop storage tradeoff¶
The bus-hop storage tradeoff is the architectural bargain a cloud operator makes when deciding whether a compute host's storage lives on the same physical (local NVMe, one bus hop away) or in a SAN fabric (network-attached block storage abstracted away from compute).
The two options¶
| Local NVMe | SAN fabric (EBS-shaped) | |
|---|---|---|
| Read latency | Bus-hop (single-digit μs) | Network-plus-SAN (low ms) |
| Build cost | Low (commodity NVMe) | High (regional SAN per region) |
| Workload mobility | Anchored to the physical | Free (any host in the region) |
| Drain operation | Hard (stateful workloads can't trivially relocate) | Easy (stateless-equivalent) |
| Single-node durability | None (node loss = data loss) | Multi-AZ durability engineered in |
Fly.io's choice and its consequences¶
From Making Machines Move:
At the heart of our platform is a systems design tradeoff about durable storage for applications. When we added storage three years ago, to support stateful apps, we built it on attached NVMe drives. A benefit: a Fly App accessing a file on a Fly Volume is never more than a bus hop away from the data. A cost: a Fly App with an attached Volume is anchored to a particular worker physical.
Fly explicitly framed this as a startup-appropriate choice:
We're a startup. Building SAN infrastructure in every region we operate in would be tremendously expensive. Look at any feature in AWS that normal people know the name of, like EC2, EBS, RDS, or S3 — there's a whole company in there. We launched storage when we were just 10 people, and even at our current size we probably have nothing resembling the resources EBS gets.
The consequence is that every architectural wrinkle in the
2024-07-30 post — the inability to drain, the need to build
dm-clone-based migrations, the temporary-
SAN fabric during drain, iSCSI, the LUKS2 header-size skew, the
6PN address rewrites — is a direct cost of the local-NVMe choice.
They are the compounded interest on the bus-hop bet.
The tradeoff re-framed by LSVD¶
The post ends by gesturing at LSVD — local NVMe as a cache in front of S3-shaped object storage — as a way to keep the bus-hop bet on the read path while removing the anchored-to-physical cost. "You get most of the performance benefit of bus-hop disk writes, along with unbounded storage and S3-grade reliability." The architectural shift is NVMe-as-cache rather than NVMe-as-durable.
Seen in¶
- sources/2024-07-30-flyio-making-machines-move — The canonical wiki phrasing of the tradeoff; Fly.io's retrospective on having picked local NVMe and dealt with the downstream consequences.
Related¶
- systems/fly-volumes — Local-NVMe instance.
- systems/aws-ebs — SAN-fabric instance.
- systems/lsvd — The hybrid cache-in-front-of-object-store resolution.
- concepts/fleet-drain-operation — The operational surface the tradeoff impacts.