Skip to content

FLYIO 2026-10-22

Read original ↗

SBD: We Have a Filesystem at Home

Summary

Fly.io replaced the storage stack behind Sprites with the Sprites Block Device (SBD): it takes an object-storage bucket and presents it to the Linux kernel as a plain block device over NBD, which the Sprite formats as ext4 and mounts like a disk on any Linux box. SBD retires the v1 stack — a "heavily modded" JuiceFS fork that stored a single 100 GB ext4 disk image as object-store chunks, with a rewritten SQLite metadata backend kept durable by Litestream, plus a second checkpoints.db. That design kept three separately-durable ledgers (chunks in the bucket, the chunk-map metadata DB, the checkpoint DB) that each "could be right while the others were wrong"; a dogfooding harness that drove 400 JuiceFS Sprites through ~5,000 write/checkpoint/sleep/wake/restore cycles left "a nonzero number" permanently unable to checkpoint. SBD collapses the ledgers to one — metadata lives in the bucket alongside the blocks it describes — so ext4 does all the file semantics and a single sbd serve daemon answers kernel block I/O from a local cache while uploading dirty blocks in the background. The result is reported as "orders of magnitude more reliable", with simpler checkpoints (a versioned block snapshot of the whole volume), restore that "costs about as much as a restart", per-block read-back verification, and fail-stop behavior (a Sprite that can't vouch for its disk stops and restarts rather than continuing).

Key takeaways

  1. The durable root is still object storage — that never changed. "The root of a Sprite's disk is object storage, not the NVMe in whatever server the Sprite happens to be running on. That's what lets a Sprite sleep, wake on a different server, and still find its files where it left them. The durable state of a Sprite is, more or less, a location in a bucket." SBD changes the translation layer, not the object-store-as-disk-root premise. (Source: this article.)

  2. The v1 problem was the translation layer, not the bucket. Object stores "can't change part of an object. To change one byte, you have to replace the whole thing" — a bad cultural fit for Linux programs that rewrite bytes in the middle of files constantly. v1 bridged this with a JuiceFS fork that "stored exactly one file: a 100 GB disk image, mounted on a loop device with ext4 inside", split into data chunks (bucket) + a rewritten-SQLite metadata map (Litestream-durable). (Source: this article.)

  3. Three ledgers that can independently diverge = the core failure mode. "A filesystem with a metadata database has two inseparable components: the chunks in the bucket, and the database that explains how those chunks fit together. When they agree, you have files. When they don't, you're screwed." Each checkpoint added more state: a copy-on-write clone tracked in the metadata DB, plus a second checkpoints.db naming which image to mount per checkpoint. "Each one was made durable separately, so each one could be right while the others were wrong." This is a storage-stack instance of the multi-ledger-divergence failure. (Source: this article.)

  4. Writeback mode turned a crash into permanent corruption. To go fast, JuiceFS ran in writeback mode: "It records a chunk in its metadata first and uploads the bytes after. If the machine is killed at any time between those two steps, the metadata points at a chunk that doesn't exist." A boot then gets a 404 for a chunk; the ext4 layer on top finds a directory block that fails its checksum and goes read-only. (Source: this article.)

  5. CoW checkpoints inherited the gap. Because JuiceFS checkpoints were clones that "shared the same chunks", a missing chunk was missing from every checkpoint — "Moving the Sprite to a new machine won't help, nor will restoring a checkpoint." Shared-chunk CoW means a single torn write corrupts the whole lineage. (Source: this article.)

  6. SBD = bucket → NBD → ext4. "It takes a bucket and presents it to the Linux kernel as a block device, over NBD. The Sprite formats that device as ext4 and mounts it, the same way you'd mount a disk on any Linux box… the metadata lives in the bucket alongside the blocks it describes, so there's no separate database to keep in sync." One daemon (sbd serve) answers kernel reads/writes from its local cache, uploads dirty blocks in the background, and "flushes on demand when something needs a guarantee that the bytes have left the building." (Source: this article.)

  7. Dumber checkpoints are better checkpoints. Under SBD a checkpoint is a block snapshot of the whole volume with a version number, recorded in SBD's own checkpoint catalog — "the unmitigated comfort of a single source of truth." Restore "unmounts the filesystem, rewinds the volume in place and mounts it again. It costs about as much as a restart." (Source: this article.)

  8. Fail-stop over fail-dirty, with per-block verification. "SBD checks every block it reads back from object storage against what it wrote. If SBD itself dies, the whole Sprite restarts, because a clean restart is the only way back to a disk you can trust." A Sprite "now stops and says so instead of carrying on with a disk it can't vouch for." (Source: this article.)

  9. Agents did the dogfooding. Fly.io built a harness that "drives fleets of Sprites through write, checkpoint, sleep, wake and restore, over and over, verifying every byte that comes back" and pointed it at 400 JuiceFS Sprites for about 5,000 cycles — the test that surfaced the permanent-checkpoint-loss bug. SBD is reported "orders of magnitude more reliable" against the same "borderline abusive" harness. (Source: this article.)

  10. Design philosophy — buy the $1 bag of spaghetti. The framing: infra should be "a simple thing that works well", not a clever bespoke filesystem that "comes with a repair kit." SBD's whole value proposition is that "the only thing keeping track of your files is the filesystem code every Linux box already runs." (Source: this article.)

Architecture / mechanism detail

v1 (JuiceFS fork) — the stack being retired

  • Chunks in an S3-compatible bucket (likely Tigris, unnamed) hold the byte volume; chunks are immutable/content-addressed.
  • Metadata DB — a rewritten SQLite backend (JuiceFS normally uses Redis/MySQL/Postgres/TiKV) mapping file → chunks → object keys, kept durable with Litestream. Stored exactly one file: a 100 GB ext4 disk image on a loop device.
  • checkpoints.db — a second SQLite DB listing checkpoints and which image to mount on the loop device for each.
  • Writeback mode — metadata recorded before bytes uploaded → crash window produces dangling chunk references → 404 on read → ext4 checksum fail → read-only filesystem.
  • Checkpoints = CoW clones sharing chunks → one lost chunk corrupts all checkpoints.

SBD — the replacement

  • Bucket presented to the kernel as a block device over NBD. The Sprite formats it ext4 and mounts it normally.
  • sbd serve daemon inside the Sprite: answers kernel block reads/writes from a local cache, uploads dirty blocks to object storage in the background, and flushes on demand when a durability barrier is needed.
  • Metadata co-located with blocks in the bucket → no separate DB to keep in sync; one daemon + one bucket replaces "a filesystem, a database and a disk image to keep in agreement."
  • Checkpoint = versioned whole-volume block snapshot recorded in SBD's own catalog. Restore = unmount → rewind volume in place → remount (≈ restart cost).
  • Read-back verification: every block read from object storage is checked against what was written.
  • Fail-stop: on a disk it can't vouch for, the Sprite stops and says so; if sbd itself dies, the whole Sprite restarts (clean restart = only trusted path back).
  • Migration note: Sprites created before 10/6 can be migrated via Fly's storage-upgrade instructions.

Operational numbers

  • 100 GB per-Sprite disk (v1 image size; unchanged default).
  • 400 JuiceFS Sprites × ~5,000 cycles in the dogfooding harness; "a nonzero number" permanently lost checkpoint capability under v1.
  • SBD reported "orders of magnitude more reliable" than v1 on the same harness.
  • Restore cost ≈ a restart.
  • Pre-10/6 Sprites require a storage upgrade to move to SBD.

Caveats / open questions

  • No absolute reliability numbers. "Orders of magnitude more reliable" and "a nonzero number" are the only quantification; no MTBF, no corruption rate, no post-SBD harness pass/fail counts.
  • Write-latency / throughput of the NBD+ext4 path is undisclosed. v1's JuiceFS write path and SBD's sbd serve cache+background-upload path aren't compared on latency; the earlier Sprites posts flagged object-store write latency as the reason LSVD-style block-over-object-store was "inadequate for a hot Postgres node" — whether SBD changes that ceiling is not addressed.
  • Block/cache internals undisclosed — block size, cache eviction policy, dirty-block batching, flush-barrier semantics, and how sbd maps the volume into object keys are not detailed.
  • NBD robustness. Fly previously abandoned NBD for Volume migration after "stuck nbd kernel threads" under network disruption (2024-07-30). This post doesn't explain what makes NBD safe here (likely host-local NBD to a co-located daemon rather than NBD across the fleet, but unstated).
  • Checkpoint retention / GC of old block snapshots and dirty-block lifecycle cadence aren't discussed.
  • What happened to Litestream/SQLite in the stack? SBD removes the separate metadata DB; the post doesn't say whether Litestream still underpins the global orchestrator (prior posts say per-org SQLite+Litestream for orchestration, a separate concern from the per-Sprite disk).

Source

Last updated · 854 distilled / 2,448 read