Skip to content

SYSTEM Cited by 1 source

Sprites Block Device (SBD)

Definition

SBD (Sprites Block Device) is Fly.io's storage stack for Sprites, announced 2026-10-22 as the replacement for the JuiceFS-fork stack that shipped Sprites at launch. SBD "takes a bucket and presents it to the Linux kernel as a block device, over NBD. The Sprite formats that device as ext4 and mounts it, the same way you'd mount a disk on any Linux box." A daemon inside the Sprite, sbd serve, answers the kernel's block reads and writes from a local cache, uploads dirty blocks to object storage in the background, and flushes on demand when something needs a durability guarantee.

SBD keeps Sprites' founding premise unchanged — the root of durability is S3-compatible object storage, not local NVMe — but swaps the translation layer: instead of a filesystem + metadata database + disk image kept in agreement, there is one daemon and one bucket, with the filesystem's own metadata living in the bucket alongside the blocks it describes. "Ext4 still does the legwork." (Source: sources/2026-10-22-flyio-sbd-we-have-a-filesystem-at-home.)

Why SBD exists — the v1 (JuiceFS) failure mode

The launch stack was a "heavily modded" JuiceFS fork that stored exactly one file: a 100 GB disk image, mounted on a loop device with ext4 inside. JuiceFS split storage into data chunks (in the bucket) and a chunk-map metadata DB (a rewritten SQLite backend kept durable with Litestream); a second checkpoints.db named which image to mount per checkpoint. That left three separately-durable ledgers:

  1. the chunks in the bucket,
  2. the metadata DB mapping chunks → the live image,
  3. the checkpoint DB mapping checkpoints → images.

"Each one was made durable separately, so each one could be right while the others were wrong." This is a storage-stack instance of the multi-ledger-divergence failure.

Two mechanisms turned divergence into permanent corruption:

  • Writeback mode. For speed, JuiceFS "records a chunk in its metadata first and uploads the bytes after. If the machine is killed at any time between those two steps, the metadata points at a chunk that doesn't exist" → a boot gets a 404 for the chunk → ext4 finds a directory block that fails its checksum → the filesystem goes read-only.
  • Copy-on-write checkpoints that share chunks. A JuiceFS checkpoint was a CoW clone sharing the live disk's chunks, so a missing chunk was missing from every checkpoint — "Moving the Sprite to a new machine won't help, nor will restoring a checkpoint… every checkpoint would have the same gap."

A dogfooding harness (see below) drove 400 JuiceFS Sprites through ~5,000 cycles and left "a nonzero number" permanently unable to checkpoint. Fly.io's conclusion: adding "more reconciliation, more repair tooling, and increasingly clever ways to keep three ledgers in agreement" was the wrong direction — "spaghetti that comes with a repair kit."

Architecture

Data path

  • Bucket → NBD → ext4. The bucket is presented to the Linux kernel as a block device via NBD; the Sprite runs stock ext4 on top. All file semantics (directories, inodes, in-place byte rewrites, atomic rename) are handled by "the filesystem code every Linux box already runs" — see concepts/file-vs-object-semantics.
  • sbd serve daemon. Lives inside the Sprite. Serves kernel block reads and writes from a local cache, streams dirty blocks to object storage in the background, and performs a flush on demand when a durability barrier is required.
  • Metadata co-located with data. The filesystem's metadata lives in the bucket next to the blocks it describes — "so there's no separate database to keep in sync." One source of truth replaces the v1 three-ledger stack.

Checkpoint / restore

  • Checkpoint = a versioned whole-volume block snapshot, recorded in SBD's own checkpoint catalog — "the unmitigated comfort of a single source of truth." (Contrast v1's CoW image clones tracked across two SQLite DBs.)
  • Restore "unmounts the filesystem, rewinds the volume in place and mounts it again. It costs about as much as a restart." Each checkpoint knows which version of the volume it belongs to.

Integrity model — fail-stop

  • Per-block read-back verification: "SBD checks every block it reads back from object storage against what it wrote."
  • Fail-stop, not fail-dirty: a Sprite "now stops and says so instead of carrying on with a disk it can't vouch for." If sbd itself dies, "the whole Sprite restarts, because a clean restart is the only way back to a disk you can trust."

Dogfooding harness

SBD's reliability claim rests on a "borderline abusive" agent-driven test harness that "drives fleets of Sprites through write, checkpoint, sleep, wake and restore, over and over, verifying every byte that comes back." It was pointed at 400 JuiceFS Sprites for ~5,000 cycles (surfacing the permanent-checkpoint-loss bug) and then at SBD, which came back "orders of magnitude more reliable than our homemade pasta filesystem." See patterns/custom-benchmarking-harness (the correctness-regression variant: a custom harness verifying every byte, not a latency benchmark).

Relationship to Fly's other storage efforts

  • v1 JuiceFS fork — the stack SBD replaces. SBD keeps the object-store root but drops the chunk-map metadata DB, the checkpoint DB, and the disk-image-on-a-loop-device indirection.
  • LSVD — Fly's earlier "bottomless S3-backed volumes" experiment, which also presented object storage as a block device, but at the Fly-Machine level; its performance ceiling was "inadequate for a hot Postgres node." SBD is the Sprites-level realization of the same block-device-over- object-store idea for workloads that don't need Postgres-grade write latency.
  • NBD — Fly previously abandoned NBD for Volume migration (2024-07-30) after stuck kernel threads under fleet-wide network disruption; SBD is a new, positive NBD use (presumably host-local NBD to the co-located sbd daemon rather than NBD across the fleet, though the post doesn't say).

Operational numbers

  • 100 GB default Sprite disk (unchanged from v1).
  • Dogfooding: 400 Sprites × ~5,000 cycles; v1 lost checkpoint capability on "a nonzero number"; SBD "orders of magnitude more reliable."
  • Restore ≈ a restart.
  • Sprites created before 10/6 require a storage upgrade to migrate to SBD.

Caveats / open questions

  • No absolute reliability figures beyond "orders of magnitude" and "a nonzero number."
  • Write-latency / throughput of the NBD+ext4+sbd path is undisclosed — and not compared to v1 or to the LSVD ceiling that ruled out object-store-rooted disks for hot Postgres.
  • Block/cache internals undisclosed: block size, cache eviction, dirty-block batching, flush-barrier semantics, object-key layout of the volume.
  • NBD robustness rationale unstated given Fly's prior abandonment of NBD.
  • Checkpoint retention / block-snapshot GC cadence not discussed.

Seen in

  • sources/2026-10-22-flyio-sbd-we-have-a-filesystem-at-home — canonical and sole source; introduces SBD, documents the v1 JuiceFS three-ledger / writeback-404 / shared-chunk-CoW failure modes, and the SBD design (bucket → NBD → ext4, sbd serve, block-snapshot checkpoints, read-back verification, fail-stop).
Last updated · 854 distilled / 2,448 read