Skip to content

Fly.io — Litestream Writable VFS

Summary

Ben Johnson's follow-up to the 2025-12-11 Litestream VFS ship post, published 2026-02-04. The post discloses two new capabilities added to the VFS — write mode and background hydration — and pins them to a concrete production consumer: Fly.io's new Sprites product uses Litestream VFS as the SQLite metadata backend for the Sprite disk stack's block map (a rewrite of the metadata tier of a forked JuiceFS, tongue-in-cheek characterized as "it's Litestream SQLite, of course" after the false BoltDB misdirect). Sprites boot in under a second, carry 100 GB of durable storage, and need the block map ready within the HTTP-request budget of the boot that provokes them — a usage pattern that broke the earlier read-only / cold-start-optimized VFS shape and drove the two new features.

The writable VFS is deliberately narrow: single-writer only, no polling for remote writers, writes go to a local temporary buffer and sync to object storage every ~1 s or on clean shutdown, durability matches the existing "eventual durability" property of Sprite storage rather than strict synchronous-to-object-store. Hydration is equally narrow: when LITESTREAM_HYDRATION_PATH is set, the VFS serves reads from object storage while a background loop pulls the full database to a local temp file (shaped like litestream restore, using LTX compaction to emit only the latest version of each page), and switches the read path over once hydration completes; on VFS exit the hydration file is discarded.

Johnson is explicit that "these features are narrowly scoped for problems that look like the ones our storage stack needs" and that "for ordinary read/write workloads, you don't need any of this mechanism." The value to the wiki is (a) the writable-VFS design contract as a documented counterexample to generic multi-writer distributed SQLite (the "Lament Configuration" is called out by name), and (b) the Sprite block-map disclosure that fills in the previously-opaque "metadata lives in fast local storage… kept durable with Litestream" phrasing from the 2026-01-14 Sprites design post with its actual shape: Litestream-VFS-over-SQLite as the JuiceFS metadata tier.

Key takeaways

  1. Two new VFS features ship together: write mode and hydration. Both are gated on environment variables (LITESTREAM_WRITE_ENABLED=true, LITESTREAM_HYDRATION_PATH=/path/to/file) and both sit alongside the read-only VFS shape disclosed in 2025-12-11. (Source: sources/2026-02-04-flyio-litestream-writable-vfs)

  2. Writable VFS assumes a single writer and disables polling. "In write mode, we don't allow multiple writers, because multiple-writer distributed SQLite databases are the Lament Configuration and we are not explorers over great vistas of pain. So the VFS in write-mode disables polling. We assume a single writer, and no additional backups to watch." This is an explicit design-decision disclosure: Litestream writable VFS is not a CRDT/multi-leader SQLite; it is a single-writer-with-buffered-sync primitive. (Source: sources/2026-02-04-flyio-litestream-writable-vfs)

  3. Writes go to a local buffer, synced to object storage every ~1 s or on clean shutdown. "Writes go to a local temporary buffer ('the write buffer'). Every second or so (or on clean shutdown), we sync the write buffer with object storage. Nothing written through the VFS is truly durable until that sync happens." The durability SLO is therefore bounded-loss — at most ~1 s of writes can be lost on unclean shutdown before they reach object storage. (Source: sources/2026-02-04-flyio-litestream-writable-vfs)

  4. The durability contract matches Sprites' existing "eventual durability" — not a general SQLite durability upgrade. "Critically, we support that use case only up to the same durability requirements that a Sprite already has. All storage on a Sprite shares this 'eventual durability' property, so the terms of the VFS write make sense here. They probably don't make sense for your application." Johnson flags this explicitly as a narrow-fit feature. (Source: sources/2026-02-04-flyio-litestream-writable-vfs)

  5. Hydration trick borrowed from dm-clone: serve remotely while pulling a full local copy in the background. "To solve this problem, we shoplifted a trick from systems like dm-clone: background hydration. In hydration designs, we serve queries remotely while running a loop to pull the whole database. When you start the VFS with the LITESTREAM_HYDRATION_PATH environment variable set, we'll hydrate to that file." First wiki citation of dm-clone hydration as conceptual ancestor of the Litestream VFS read path — connects the 2024-07-30 Making Machines Move post's block-level lineage. (Source: sources/2026-02-04-flyio-litestream-writable-vfs)

  6. Hydration writes only the latest version of each page using LTX compaction. "Hydration takes advantage of LTX compaction, writing only the latest versions of each page. Reads don't block on hydration; we serve them from object storage immediately, and switch over to the hydration file when it's ready." The hydration file is "simply a full copy of your database. It's the same thing you get if you run litestream restore." (Source: sources/2026-02-04-flyio-litestream-writable-vfs)

  7. Hydration file is ephemeral by design — discarded on VFS exit. "Because this is designed for environments like Sprites, which bounce a lot, we write the database to a temporary file. We can't trust that the database is using the latest state every time we start up, not without doing a full restore, so we just chuck the hydration file when we exit the VFS. That behavior is baked into the VFS right now." Trade-off explicitly flagged as Sprite-fit, not general. (Source: sources/2026-02-04-flyio-litestream-writable-vfs)

  8. Sprites are a load-bearing Litestream consumer, in two ways. "Sprites rely directly on Litestream in two big ways. First, Litestream SQLite is the core of our global Sprites orchestrator… Every organization enrolled in Sprites gets their own SQLite database, synchronized by Litestream… The second way Sprites use Litestream is much more interesting. Litestream is built directly into the disk storage stack that runs on every Sprite." Per-org Litestream DB for the global orchestrator (replacing the Fly Machines' centralized Postgres); per-Sprite Litestream VFS for the in-VM block map. (Source: sources/2026-02-04-flyio-litestream-writable-vfs)

  9. The Sprite block map is the JuiceFS metadata tier, rewritten on SQLite+Litestream. "The system that does this is JuiceFS, and the database — let's call it 'the block map' — is a rewritten metadata store, based (you guessed it) on BoltDB. I kid! It's Litestream SQLite, of course." Block maps are "not huge, but they're not tiny; maybe low tens of megabytes worst case." This fills in the previously-vague "metadata lives in fast local storage… kept durable with Litestream" phrasing from the 2026-01-14 Sprites design post. (Sources: sources/2026-02-04-flyio-litestream-writable-vfs, sources/2026-01-14-flyio-the-design-implementation-of-sprites)

  10. The driving latency budget is "HTTP-request-timely Sprite boot." "If the Fly Machine underneath a Sprite bounces, we might need to reconstitute the block map from object storage. Block maps aren't huge, but they're not tiny; maybe low tens of megabytes worst case. The thing is, this is happening while the Sprite boots back up… that's something that can happen in response to an incoming web request; that is, we have to finish fast enough to generate a timely response to that request. The time budget is small." This ties the "fast startup for ephemeral servers" claim from the 2025-12-11 post to a specific production latency requirement. (Source: sources/2026-02-04-flyio-litestream-writable-vfs)

  11. "Run queries before downloading the database" is the headline capability both features defend. "Litestream VFS lets us run point-in-time SQLite queries hot off object storage blobs, answering queries before we've downloaded the database." The writable VFS removes the read-only constraint that blocked Sprite storage-stack use; hydration removes the steady-state latency-tax of range-GET-per-page. (Source: sources/2026-02-04-flyio-litestream-writable-vfs)

  12. Opt-in, feature-flagged, additive. The new capabilities are reached only via environment variables (LITESTREAM_WRITE_ENABLED, LITESTREAM_HYDRATION_PATH). The default VFS shape is unchanged — still the 2025-12-11 read-only near-realtime-replica with LRU + Range GET + L0 polling. "For ordinary read/write workloads, you don't need any of this mechanism." (Source: sources/2026-02-04-flyio-litestream-writable-vfs)

Systems introduced

Systems extended

  • systems/litestream-vfs — new write mode + hydration mode capabilities documented; frontmatter and body extended.
  • systems/litestream — follow-up ship post added to sources list; body updated with 2026-02-04 writable/hydration disclosure under a new subsection of the VFS section.
  • systems/fly-sprites — block map identified as Litestream-VFS-backed SQLite; orchestrator per-org-DB shape documented; two-axis Litestream usage (orchestrator + storage-stack) captured.
  • systems/juicefs — metadata-backend rewrite identified as Litestream-VFS SQLite (previously documented as "rewritten SQLite metadata backend" without the Litestream-VFS tie-in).
  • systems/dm-clone — "hydration design" terminology cross-referenced as the conceptual ancestor of the Litestream VFS hydration mode (prior wiki citation framed dm-clone only in the block-level-async-clone / Fly-Volumes migration context).

Concepts introduced

Concepts extended

  • concepts/sqlite-vfs — frontmatter sources extended; new body subsection for the 2026-02-04 writable-VFS + hydration-mode disclosures (the VFS interface now intercepts writes too, and adds a background-hydrate surface).
  • concepts/async-clone-hydration — this article's first wiki citation of "hydration design" applied at the-SQLite-database-as-unit level (previous instances were block-level via dm-clone and dm-cache). Frontmatter + body extended.

Patterns introduced

  • None net-new — the two new VFS capabilities fit inside existing patterns:
  • Write-mode sits inside the patterns/vfs-range-get-from-object-store pattern (now with a bidirectional variant), but doesn't generalise as a standalone pattern (single-writer-only, Sprite-fit only, per the post).
  • Hydration is a specific shape of patterns/read-through-object-store-volume at the database-page level instead of the block level — the conceptual shape (serve-remote-while-pulling-local-in-background) is the dm-clone lineage the post explicitly names. Captured via an extension to read-through-object-store-volume rather than a new pattern page, since the mechanics are the same at a different granularity.

Patterns extended

  • patterns/vfs-range-get-from-object-store — frontmatter sources extended; body extended with the 2026-02-04 write-mode variant (single-writer, ~1 s buffer-sync, eventual-durability SLO).
  • patterns/near-realtime-replica-via-l0-polling — brief extension: hydration-mode interacts with L0-polling (hydration fills the local file, polling continues against the upstream L0 stream; documented as a note).
  • patterns/read-through-object-store-volume — extended with the database-level-hydration variant (Litestream VFS hydration file is the DB equivalent of the NVMe-sparse read-through cache in front of a JuiceFS volume).
  • patterns/metadata-plus-chunk-storage-stack — Fly Sprites block map usage captured as the first production wiki instance where the metadata tier in a metadata-plus-chunks stack is itself a Litestream-VFS-over-SQLite backed by object storage. The metadata tier is itself object-store-rooted (recursive application of the pattern).

Operational numbers

Item Value Source
Block map size (worst case) "low tens of megabytes" 2026-02-04 post
Writable-VFS buffer-sync cadence ~1 s (or clean shutdown) 2026-02-04 post
Writers supported by writable VFS 1 (single-writer only, polling disabled) 2026-02-04 post
Hydration file lifetime ephemeral — discarded on VFS exit 2026-02-04 post
Hydration read-blocking none — reads served from object storage during hydration 2026-02-04 post
Hydration compaction latest-version-per-page only (LTX compaction) 2026-02-04 post
Activation LITESTREAM_WRITE_ENABLED=true; LITESTREAM_HYDRATION_PATH=/path/to/file 2026-02-04 post
Sprite boot budget "in response to an incoming web request" (HTTP-request-timely) 2026-02-04 post
Sprite default storage 100 GB durable 2026-02-04 post (reiteration)
Sprite boot latency "under a second" 2026-02-04 post (reiteration)
Sprites-orchestrator DB topology 1 SQLite DB per organization, Litestream-synced 2026-02-04 post

Caveats

  • Writable VFS is Sprite-fit, explicitly not general. The post repeats the warning three times — "not as general-purpose as it might look", "they probably don't make sense for your application", "narrowly scoped for problems that look like the ones our storage stack needs". The wiki captures this as a documented narrow-fit feature, not a new default.
  • No durability-loss-window SLO disclosed beyond "every second or so." Post does not pin a maximum loss window under unclean shutdown. Treat ~1 s as an upper bound with unknown jitter.
  • No concurrent-writer coordination disclosed — the VFS assumes single-writer but the post doesn't discuss how Sprites enforce that assumption (lease? machine-count=1? single-process invariant on the Sprite VM?). The CASAAS machinery Litestream inherited from the 2025-05-20 redesign is the most likely enforcement primitive but not explicitly invoked here.
  • Hydration-discard-on-exit has no opt-out knob disclosed. "That behavior is baked into the VFS right now." Readers who want persistent post-hydration local DB files need to wait for configurability or accept re-hydration on every VFS startup.
  • No Sprite-level perf numbers for block-map reconstitute-during-boot. The post frames the budget qualitatively ("the time budget is small", "fast enough to generate a timely response") but discloses no concrete p50/p99 latencies for the block-map-from-object-storage cold path. The 2025-12-11 post's "fast startup" framing is reiterated qualitatively.
  • Point-in-time read (PRAGMA litestream_time) + write-mode interaction undisclosed. The 2025-12-11 post introduced SQL-level PITR via PRAGMA litestream_time; the 2026-02-04 post doesn't say what LITESTREAM_WRITE_ENABLED does to point-in-time queries (presumably disables them, but not confirmed).
  • Hydration + write-mode interaction undisclosed. The post describes hydration separately from write mode. Whether both can coexist on a single VFS connection (write-buffer-on-top-of- hydrated-local-file) is not discussed. The Sprite block map apparently uses both, but the coexistence semantics aren't pinned.
  • Per-org orchestrator DB scaling breakdown undisclosed. Orchestrator is "global" and per-org SQLite; throughput per DB, number of orgs enrolled in Sprites, cross-org queries (if any), and failure-domain boundaries are all undisclosed. The post frames the shape as "this is a fun design" with "nice scaling characteristics" but stops short of numbers.
  • No comparison to Fly Machines' Postgres cluster. The post observes that the Sprite orchestrator "runs directly off S3-compatible object storage" — "unlike our flagship Fly Machines product, which relies on a centralized Postgres cluster" — but doesn't quantify the operational load difference.

Source

Last updated · 319 distilled / 1,201 read