Skip to content

FLYIO 2026-01-29 Tier 3

Read original ↗

Fly.io — Litestream Writable VFS

Summary

Ben Johnson's 2026-01-29 shipping post announcing two new opt-in modes for Litestream VFS — the SQLite VFS extension shipped 2025-12-11 — driven directly by the needs of the Fly.io Sprites storage stack: (1) an optionally-writable mode (LITESTREAM_WRITE_ENABLED=true) that enables writes against the VFS-backed database with a local temporary write buffer synced to object storage "every second or so (or on clean shutdown)"; and (2) a background hydration mode (LITESTREAM_HYDRATION_PATH=...) that serves queries remotely while a background loop downloads the full database to a local file and then switches reads over. The Sprites-motivated use case is the block-map metadata store inside every Sprite's disk stack — a "hacked-up JuiceFS, with a rewritten SQLite metadata backend" (already known from the 2026-01-14 Sprites design post) whose metadata DB is "low tens of megabytes worst case" and must serve writes milliseconds after a Sprite boots from object storage. The writable-VFS mode is deliberately scoped to exactly the durability class Sprites already have — eventual durability — and is explicitly not pitched as a general-purpose multi-writer distributed-SQLite surface: "multiple-writer distributed SQLite databases are the Lament Configuration and we are not explorers over great vistas of pain." The writable VFS disables the L0 polling loop (that's the read-mode near-realtime-replica behaviour from the 2025-12-11 post) and assumes a single writer because writes are buffered locally and there's no need to watch for remote-writer LTX files. Hydration is shoplifted from dm-clone: reads don't block on hydration (served from object storage via Range GETs immediately); the VFS switches reads over to the hydration file once it's complete. Hydration exploits LTX compaction to write "only the latest versions of each page" — same cost structure as a litestream restore. The hydration file is written to a temp file discarded on exit because the VFS can't trust a prior hydration file reflects the latest state on a Sprite bounce. Closing framing positions both features as narrowly scoped for Sprites' needs — "features are narrowly scoped for problems that look like the ones our storage stack needs" — with explicit invitation for other uses "if for some reason they do, have at it!" Sprites' other Litestream use — the global orchestrator's per-org Elixir/SQLite DB — is "boring" and not covered here.

Key takeaways

  1. Writable VFS mode is opt-in via environment variable. "To enable writes with Litestream VFS, just set the LITESTREAM_WRITE_ENABLED environment variable true." The same VFS extension (.load litestream.so + file:///my.db?vfs=litestream) can be used read-only (default) or read-write. (Source: sources/2026-01-29-flyio-litestream-writable-vfs)

  2. Writes go to a local write buffer, synced to object storage every ~1s or on clean shutdown. "Writes go to a local temporary buffer ('the write buffer'). Every second or so (or on clean shutdown), we sync the write buffer with object storage. Nothing written through the VFS is truly durable until that sync happens." This is the eventual durability contract exactly — same window a Sprite's other storage operates under. Canonical instance of patterns/writable-vfs-with-buffered-sync.

  3. Single-writer assumption disables polling. "In write mode, we don't allow multiple writers, because multiple-writer distributed SQLite databases are the Lament Configuration and we are not explorers over great vistas of pain. So the VFS in write-mode disables polling. We assume a single writer, and no additional backups to watch." The L0-polling behaviour from the read-only VFS (patterns/near-realtime-replica-via-l0-polling) is turned off in write mode — there's no remote writer whose LTX emissions the reader needs to see.

  4. Motivating use case: Sprite block-map on cold boot. "A Sprite is cold-starting and its storage stack needs to serve writes, milliseconds after booting, without having a full copy of the 10MB block map. This writeable VFS mode lets us do that." The block map is the JuiceFS-lineage metadata store tracking which chunks live where; it must be writable from request-handler-time latency budgets on a Sprite bounce, which the VFS mode serves.

  5. Durability contract matches the Sprite substrate. "We support that use case only up to the same durability requirements that a Sprite already has. All storage on a Sprite shares this 'eventual durability' property, so the terms of the VFS write make sense here." Explicit framing: the writable-VFS durability class is intentionally Sprite-shaped and the post warns other applications: "They probably don't make sense for your application."

  6. Hydration ships a background-fill loop alongside remote reads. "In hydration designs, we serve queries remotely while running a loop to pull the whole database. When you start the VFS with the LITESTREAM_HYDRATION_PATH environment variable set, we'll hydrate to that file. Hydration takes advantage of LTX compaction, writing only the latest versions of each page. Reads don't block on hydration; we serve them from object storage immediately, and switch over to the hydration file when it's ready." The prior-art callout: "we shoplifted a trick from systems like dm-clone: background hydration." Canonical instance of patterns/background-hydration-to-local-file.

  7. Hydration file is disposable-per-run. "We can't trust that the database is using the latest state every time we start up, not without doing a full restore, so we just chuck the hydration file when we exit the VFS. That behavior is baked into the VFS right now." On Sprite-bounce-heavy environments the safe default is re-hydrate-from-scratch — skipping a full litestream restore on the cold path but not assuming any prior hydration file is trustworthy. Temp-file + discard-on- exit is load-bearing for the correctness argument.

  8. Both features are narrowly scoped for Sprites. "the features are narrowly scoped for problems that look like the ones our storage stack needs. If you think you can get use out of them, I'm thrilled, and I hope you'll tell me about it." Ordinary read/write SQLite workloads do not need any of this — "Litestream works fine without the VFS, with unmodified applications, just running as a sidecar alongside your application. The whole point of that configuration is to efficiently keep up with writes." Writable-VFS and hydration only pay off when cold-open latency matters and the local sidecar-mode assumptions break.

  9. Sprites have two distinct Litestream deployments. The post discloses both: (i) the global Sprites orchestrator runs "directly off S3-compatible object storage" with per-org SQLite databases synchronized by Litestream ("unlike our flagship Fly Machines product, which relies on a centralized Postgres cluster") — "boring", standard Litestream usage; (ii) Litestream "is built directly into the disk storage stack that runs on every Sprite" — the block-map backend inside every Sprite's JuiceFS stack. The writable-VFS work is for (ii); (i) is mentioned only as context.

  10. Block-map size disclosed: "low tens of megabytes worst case." "Block maps aren't huge, but they're not tiny; maybe low tens of megabytes worst case." Small enough that writable-VFS-with-buffered-sync is feasible (not a terabyte-scale database), large enough that cold-downloading the whole thing before serving the first request blows the millisecond budget.

Architectural details

Read-only VFS (2025-12-11 baseline)

Recap of the baseline the new modes build on (canonicalised on systems/litestream-vfs and sources/2025-12-11-flyio-litestream-vfs):

  • Load: .load litestream.so + file:///my.db?vfs=litestream.
  • Override read-side only; writes flow through the regular Litestream Unix program.
  • Page lookup via LTX end-of-file index trailers (~1% of each LTX file).
  • Page reads via HTTP Range GET against object storage (patterns/vfs-range-get-from-object-store).
  • LRU cache fronts Range GETs.
  • Poll L0 (1-file-per-second LTX uploads) for near-realtime replica behaviour (patterns/near-realtime-replica-via-l0-polling).
  • PRAGMA litestream_time = '<timestamp>' for SQL-level PITR.

Writable VFS mode (2026-01-29)

New knobs + new control flow:

LITESTREAM_WRITE_ENABLED=true
  • L0 polling off (no remote writer to observe).
  • Writes land in a local write buffer (temp file).
  • Every ~1s (and on clean shutdown): write buffer syncs to object storage as new LTX files.
  • Reads still resolve via the VFS page index + Range GETs against object storage, composited with the local write-buffer for uncommitted-to-remote pages.

Durability window: a page written just now is durable after the next sync tick, not immediately. Crash before sync = lose writes since last sync. Explicitly matches Sprites' eventual-durability envelope.

Background hydration mode (2026-01-29)

New knob + new control flow:

LITESTREAM_HYDRATION_PATH=/path/to/hydrated.db
  • VFS starts serving reads via Range GETs against object storage immediately (cold-open unchanged).
  • Background thread pulls LTX files, reconstructs the full database, writes the target hydration file using compaction ("only the latest versions of each page").
  • When hydration is complete, VFS switches reads over to the local file — no longer pays Range-GET round-trips.
  • On process exit, the hydration file is deleted.
  • "The hydration file is simply a full copy of your database. It's the same thing you get if you run litestream restore."

Prior-art: dm-clone — Linux device-mapper target that clones a remote block device in the background while serving reads from the remote source until the clone catches up. Same architectural shape at a different substrate.

Reads don't block on hydration. Writes during hydration (when write mode is also enabled) flow through the write buffer as normal.

Composition of both modes

The post doesn't state it explicitly, but the Sprite block-map case implies both flags used together: Sprite cold-boot starts the VFS with LITESTREAM_WRITE_ENABLED=true (so writes can happen immediately) and LITESTREAM_HYDRATION_PATH=... (so steady-state reads aren't paying S3 round-trips forever). Cold start: write-buffer active + hydration running in the background; when hydration finishes, reads transition to the local file; writes continue to flow through the write buffer.

Two phases:

  1. Cold / hydrating: reads via Range GET (slow-ish), writes via local buffer (fast) with async object-store sync.
  2. Hydrated: reads via local file (fast), writes via local buffer + async object-store sync (unchanged).

Steady-state is indistinguishable from a local SQLite database with Litestream running as a sidecar — which is the shape the closing paragraph calls out as "Litestream works fine without the VFS, with unmodified applications."

Sprites storage stack disclosure (refinement)

The 2026-01-14 Sprites design post had already disclosed:

  • Sprites use a "very hacked-up JuiceFS, with a rewritten SQLite metadata backend."
  • "That metadata store is kept durable with Litestream."
  • Sparse NVMe as dm-cache-style read-through cache.

This post adds:

  • The metadata store specifically is "the block map" — a map of (file, chunks → object-store keys).
  • Block-map size: "low tens of megabytes worst case."
  • On Fly-Machine-underneath-Sprite bounce, the block map may need to be "reconstituted from object storage."
  • Reconstitution happens "while the Sprite boots back up" — during the time budget of an incoming web request (Sprites are Anycast-addressed and can be request-triggered via Fly Proxy's autostart).
  • Writable VFS + hydration are the mechanism that makes reconstitution feasible within that budget.

Numbers disclosed

Datum Value
Write-buffer sync cadence ~1 second (and on clean shutdown)
L0 polling in write mode disabled
Block-map size (worst case) low tens of megabytes
Writer count supported in write mode 1
Hydration file persistence discarded on process exit
Durability class eventual (matches Sprite substrate)

Numbers not disclosed

  • Write-buffer size ceiling (in-memory? disk-backed? what happens if writes outrun 1-second sync cadence?).
  • Sync-ack semantics (fsync() on S3 write? eventual consistency? what does "truly durable" mean timing- wise?).
  • Hydration throughput numbers (MB/s from object storage, expected wall time for the "low tens of megabytes" block map).
  • Read-cutover mechanics (how the VFS detects hydration- complete, mid-query safety).
  • Behaviour on write-buffer-loss during crash (how much work is lost? how does the application detect it?).
  • Sprite-side block-map workload profile (reads vs writes, QPS, typical transaction size).
  • Cold-boot wall-time budget for block-map availability ("milliseconds" is directional, not a specific number).
  • Whether the writable-VFS can be composed with CASAAS (PUT-If-None-Match conditional-write lease) to make multi-writer safe (post flatly rules out multi-writer).
  • Interaction with PRAGMA litestream_time (can you rewind a writable-mode VFS? Probably not — implied by single-writer + disable-polling).
  • How write mode interacts with the L0 compaction ladder (does the writer emit directly to L0? To L1? Is there a separate write path for VFS-mode vs sidecar-mode?).

Caveats

  • Shipping post / product-announcement voice. Not an architecture paper; no formal durability proof, no benchmarking graphs, no comparison with alternatives (LiteFS-write-mode, Turso, rqlite, ChiselStore, etc.).
  • Sprite-shaped durability is a specific choice. The writable VFS is "eventual durability" — that is explicitly not suitable for applications that require strong write durability (financial systems, anything that needs fsync-to-the-wire). The post is careful to flag this multiple times.
  • Single-writer only. Any fan-out-writer architecture needs a different mechanism. CASAAS exists at Litestream-the-Unix-program but is not composed with the writable VFS here.
  • Hydration file is always discarded. Some workloads would benefit from keeping the hydration file across restarts — but the post says that behaviour is "baked into the VFS right now." Not configurable.
  • Same environment-variable configuration surface as read mode. Composable with LITESTREAM_WRITE_ENABLED and LITESTREAM_HYDRATION_PATH, but the post doesn't enumerate every combination or precedence rule.
  • No LTX-level mechanics disclosed for writes. Does the write buffer emit normal LTX files? Different file format? Does the writer participate in the L0/L1/L2/L3 compaction ladder? Post is silent.
  • Sprites-block-map deployment not yet in production at this post's publish date. "we are integrating Litestream VFS to improve start times" — present-tense integration, not past-tense disclosure. The block-map deployment may not yet be the load-bearing production substrate described in the 2026-01-14 Sprites post (which said Litestream-backed metadata, but didn't disclose which Litestream mode).

Relationship to existing wiki

Source

Last updated · 319 distilled / 1,201 read