Skip to content

SYSTEM Cited by 4 sources

Litestream VFS

Litestream VFS is a SQLite Virtual Filesystem extension that lets an unmodified SQLite library read a database "hot off an object storage URL" — individual page reads resolve to HTTP Range GETs against LTX files in S3-compatible storage, fronted by an in-process LRU cache of hot pages. Shipped in 2025-12-11 after being teased as proof-of-concept in the [[sources/2025-05-20-flyio-litestream-revamped|2025-05-20 design post]] and explicitly flagged as "not yet shipped" in the [[sources/2025-10-02-flyio-litestream-v050-is-here|2025-10-02 v0.5.0 shipping post]].

Activation surface

Standard SQLite loadable-extension mechanism:

$ sqlite3
SQLite version 3.50.4 ...
sqlite> .load litestream.so
sqlite> .open file:///my.db?vfs=litestream

The vfs=litestream URI parameter tells SQLite to route OS-level I/O for this connection through the Litestream VFS extension rather than the default OS VFS. From the application's point of view, it is still stock SQLite.

"In particular: Litestream VFS doesn't replace the SQLite library you're already using. It's not a new 'version' of SQLite. It's just a plugin for the SQLite you're already using." (Source: sources/2025-12-11-flyio-litestream-vfs)

Read-side-only

From the post:

"We override only the few methods we care about. Litestream VFS handles only the read side of SQLite. Litestream itself, running as a normal Unix program, still handles the 'write' side. So our VFS subclasses just enough to find LTX backups and issue queries."

Writes still flow through the regular Litestream primary (the Unix program running alongside the application, tailing the WAL and emitting LTX files). The VFS is an additive read-side capability, not a replacement for the write path.

Page lookup via LTX index trailer

SQLite's Read() call carries a byte offset computed against the "local file" illusion. The VFS discards that offset and uses the page number + page size to look up the remote LTX file containing the page + the real byte offset within that file. The lookup table is built from the LTX end-of-file index trailers — each LTX file's trailer is ~1% of the file size, so only a small fraction of remote bytes need to be fetched to build a database- wide page index.

"LTX trailers include a small index tracking the offset of each page in the file. By fetching only these index trailers from the LTX files we're working with (each occupies about 1% of its LTX file), we can build a lookup table of every page in the database."

Range GET against object storage

Once the page's (filename, byte offset, size) is known, the VFS issues an HTTP Range GET against the object store (S3 / Tigris / GCS / Azure Blob — any provider that honors the Range request header):

"That's enough for us to use the S3 API's Range header handling to download exactly the block we want."

Canonical instance of patterns/vfs-range-get-from-object-store.

LRU cache exploits SQLite B-tree hot-set

To avoid round-tripping to S3 on every page read, the VFS keeps an in-process LRU cache of recently-read pages. The post names the SQLite-specific hot-set shape:

"To save lots of S3 calls, Litestream VFS implements an LRU cache. Most databases have a small set of 'hot' pages — inner branch pages or the leftmost leaf pages for tables with an auto-incrementing ID field. So only a small percentage of the database is updated and queried regularly."

This is a sibling observation to the 2025-10-02 "sandwiches" worked example — AUTOINCREMENT primary keys mean every insert hits the same rightmost leaf, and queries walk the same inner branch pages to find rows; the resulting hot set is tiny and high-LRU-value.

Near-realtime replica via L0 polling

Because LTX compaction's L0 level uploads one file per second (kept only until the next L1 compaction), the VFS can poll the object-store path for new L0 files and incrementally update its page-lookup table:

"Because Litestream backs up (into the L0 layer) once per second, the VFS code can simply poll the S3 path, and then incrementally update its index. The result is a near-realtime replica. Better still, you don't need to stream the whole database back to your machine before you use it."

Canonical instance of patterns/near-realtime-replica-via-l0-polling.

SQL-level PITR via PRAGMA litestream_time

Point-in-time recovery is now a query-time knob, not a CLI operation:

sqlite> PRAGMA litestream_time = '5 minutes ago';
sqlite> SELECT * FROM sandwich_ratings ORDER BY RANDOM() LIMIT 3;
30|Meatball|Los Angeles|5
33|Ham & Swiss|Los Angeles|2
163|Chicken Shawarma Wrap|Detroit|5

Accepts relative timestamps ("5 minutes ago") or absolute ones (2000-01-01T00:00:00Z). Subsequent reads on that connection resolve against the LTX state at the chosen timestamp. Canonical instance of concepts/pragma-based-pitr.

Worked disaster-recovery example from the post: somebody runs UPDATE sandwich_ratings SET stars = 1 in prod (missing WHERE clause); on a dev machine the operator sets PRAGMA litestream_time = '5 minutes ago' and queries the table at its pre-disaster state. "Updating your database state to where it was an hour (or day, or week) ago is just a matter of adjusting the LTX indices Litestream manages."

Fast startup for ephemeral servers

"It starts up really fast! We're living an age of increasingly ephemeral servers, what with the AIs and the agents and the clouds and the hoyvin-glavins. Wherever you find yourself, if your database is backed up to object storage with Litestream, you're always in a place where you can quickly issue a query."

The cold-open path:

  1. Open connection with vfs=litestream.
  2. Fetch EOF index trailers for relevant LTX files (~1% of each).
  3. Build in-memory page index.
  4. Serve queries via cache + Range GET on misses.

No full-database download, no replica-seeding delay, no local WAL machinery. This is what makes the VFS a plausible primitive for agentic coding platforms (Phoenix.new is explicitly the kind of consumer cited across the Litestream redesign series) where per-session compute is ephemeral.

2026-02-04: Write mode and hydration

Follow-up post sources/2026-02-04-flyio-litestream-writable-vfs ships two additional capabilities, both gated on environment variables, both deliberately narrow-fit for Fly Sprites' storage stack.

Writable VFS (LITESTREAM_WRITE_ENABLED=true)

The 2025-12-11 VFS was read-only by design. 2026-02-04 adds an optional write mode:

  • Single-writer only. Polling for remote writers is disabled — the VFS assumes nothing else is writing to the same database. From the post: "multiple-writer distributed SQLite databases are the Lament Configuration and we are not explorers over great vistas of pain."
  • Local write buffer, periodic sync. "Writes go to a local temporary buffer ('the write buffer'). Every second or so (or on clean shutdown), we sync the write buffer with object storage. Nothing written through the VFS is truly durable until that sync happens."
  • Durability contract matches Sprites' "eventual durability" — not a general SQLite durability upgrade. Johnson explicitly flags the feature as narrow-fit: "they probably don't make sense for your application."

In effect: a stock SQLite connection with vfs=litestream + LITESTREAM_WRITE_ENABLED=true becomes a single-writer-with-~1s-buffered-sync SQLite against an object-store root — the same shape Sprites' in-VM storage stack already offered, now accessible at the SQLite level.

Hydration (LITESTREAM_HYDRATION_PATH=/path/to/file)

"We shoplifted a trick from systems like dm-clone: background hydration." (First wiki tie-in between Litestream VFS and the dm-clone lineage the 2024-07-30 Making Machines Move post covered.)

Mechanism:

  1. Queries are served against object storage (Range GETs + LRU cache) from VFS-open, same as the 2025-12-11 read path.
  2. A background loop litestream restores the whole database to LITESTREAM_HYDRATION_PATH, using LTX compaction to emit only the latest version of each page.
  3. When hydration completes, the read path switches over to the local file. "Reads don't block on hydration; we serve them from object storage immediately, and switch over to the hydration file when it's ready."
  4. On VFS exit, the hydration file is discarded. "We can't trust that the database is using the latest state every time we start up, not without doing a full restore, so we just chuck the hydration file when we exit the VFS."

This is the database-level sibling of Sprites' NVMe read-through cache in front of JuiceFS — same shape (serve-remote-while-populating-local-in-background), different granularity (SQLite pages vs. filesystem blocks). Canonical instance of the async-clone + background hydration pattern applied at SQLite-database granularity.

Production consumer: the Sprite block map

"Litestream is built directly into the disk storage stack that runs on every Sprite." Specifically: Fly.io's Sprite storage stack is a fork of JuiceFS with the metadata tier rewritten on SQLite + Litestream VFS — the "block map":

  • Block maps are "not huge, but they're not tiny; maybe low tens of megabytes worst case."
  • On cold boot the block map must be reconstituted "while the Sprite boots back up… that's something that can happen in response to an incoming web request; that is, we have to finish fast enough to generate a timely response to that request. The time budget is small."
  • Read-only Range-GET-per-page VFS cleared cold-boot but was "not fast enough for steady state".
  • Writable VFS + hydration cover both: cold-boot serves reads off object storage within HTTP-request latency, steady state runs against the hydrated local file with writes buffered and synced every ~1 s.

This is the concrete production use case the 2025-12-11 post's "fast startup for ephemeral servers" framing pointed at — now with a specific budget ("incoming HTTP request") and a specific workload shape (JuiceFS metadata backend = block map). The Sprite storage stack is the canonical wiki instance of the writable-VFS + hydration combination.

Concurrent use of write mode and hydration

The post describes the two features separately. Whether they coexist cleanly on a single VFS connection (write-buffer-on-top-of-hydrated-local-file) is not explicitly pinned, though the Sprite block-map usage appears to rely on both. Treat coexistence semantics as undisclosed pending further posts.

Relationship to the rest of Litestream

Opt-in, additive:

"you don't have to use our VFS library to use Litestream, or to get the other benefits of the new LTX code."

Litestream without the VFS is still the 2025-10-02 v0.5.0 system — full database restores, hierarchical compaction, NATS JetStream / S3 / GCS / Azure replica types, etc. The VFS adds a page-granular read path, near-realtime-replica behaviour, and SQL-level PITR; it doesn't replace or require anything else.

Convergence with LiteFS

LiteFS has shipped LiteVFS — its FUSE-free integration surface — since well before 2025. LiteFS-side replicas therefore already had a VFS option. With Litestream VFS shipping, the three-layer stack has a second VFS surface on the Litestream side, using the same on-disk LTX format. The architectural convergence foreshadowed in the 2025-05-20 post is now concrete: both tools share the LTX format and a VFS integration surface.

Seen in

  • sources/2025-05-20-flyio-litestream-revamped — design post. VFS-based read replicas teased as "what we're doing next", with FUSE-as-usability-wall argument.
  • sources/2025-10-02-flyio-litestream-v050-is-here — shipping post for v0.5.0. VFS explicitly flagged as "we already have a proof of concept working and we're excited to show it off when it's ready!" — not shipped in v0.5.0. The per-page compression + EOF index that v0.5.0 did ship is the structural precondition Litestream VFS needs.
  • sources/2025-12-11-flyio-litestream-vfs — ship announcement. Canonical wiki introduction of the Litestream VFS system. .load litestream.so + file:///my.db?vfs=litestream activation; page lookup via EOF-index trailers; HTTP Range GET to S3-compatible storage; LRU cache of hot pages; L0 polling for near-realtime replica; PRAGMA litestream_time for SQL-level PITR; read-side-only; opt-in.
  • sources/2026-01-29-flyio-litestream-writable-vfs — writable-VFS + hydration ship post. Adds LITESTREAM_WRITE_ENABLED (single-writer, ~1 s buffered sync, Sprites-fit eventual-durability) and LITESTREAM_HYDRATION_PATH (background-populate-local-file while serving remote reads; DB-level sibling of dm-clone-style block-level hydration). Production consumer disclosed: the Sprite block map — Fly.io's JuiceFS fork uses Litestream-VFS-over-SQLite as the metadata backend, with the HTTP-request-timely boot budget as the gating constraint.

2026-01-29 update: writable mode + hydration

On 2026-01-29 Ben Johnson published a follow-up post shipping two new opt-in modes driven by the Fly.io Sprites storage stack's needs. Source: sources/2026-01-29-flyio-litestream-writable-vfs.

Writable VFS mode

Activation:

LITESTREAM_WRITE_ENABLED=true

Enables writes against the VFS-backed database. Mechanism:

  1. Writes land in a local temporary write buffer (not object storage directly).
  2. Every ~1 second (and on clean shutdown), the write buffer syncs to object storage as new LTX files.
  3. L0 polling is disabled — there is no remote writer to observe under the single-writer assumption.

Durability class: concepts/eventual-durability — writes are not "truly durable" until the next sync completes. Explicitly scoped to match the Sprite substrate's durability envelope; explicitly not pitched as a general-purpose multi-writer distributed-SQLite surface:

"In write mode, we don't allow multiple writers, because multiple-writer distributed SQLite databases are the Lament Configuration and we are not explorers over great vistas of pain."

Canonical instance of patterns/writable-vfs-with-buffered-sync and concepts/single-writer-assumption.

Background hydration mode

Activation:

LITESTREAM_HYDRATION_PATH=/path/to/hydrated.db

While serving reads via the usual VFS Range-GET path, a background loop pulls the whole database into the specified local file, using LTX compaction to write "only the latest versions of each page". Reads do not block on hydration; the VFS switches reads over to the local file once it's ready.

Prior-art lineage: dm-clone — Linux device-mapper target that clones a remote block device in the background while serving reads from the remote source until the clone catches up.

The hydration file is a temp file discarded on process exit — the VFS can't trust a prior hydration file reflects the latest remote state on a restart (Sprites bounce a lot), so the safe default is to re-hydrate from scratch. Not configurable today: "That behavior is baked into the VFS right now."

Canonical instance of patterns/background-hydration-to-local-file and concepts/background-hydration.

Why the two modes compose for Sprites

Fly.io Sprites' disk stack uses a JuiceFS fork with a SQLite metadata backend ("the block map") kept durable via Litestream. On a Sprite bounce, the block map may need to be reconstituted from object storage — "maybe low tens of megabytes worst case" — while the Sprite boots back up in response to an incoming web request.

Vanilla litestream restore blows the millisecond-scale boot-to-first-write budget. Writable VFS + hydration together meet it: writes are served from the write buffer immediately (cold path), reads go through Range GETs on object storage immediately (cold path), and hydration warms the local read path in the background. Once hydrated, the VFS is indistinguishable from a local SQLite + sidecar Litestream deployment for the steady-state path.

Relationship to read-only VFS mode

Both new features are opt-in, additive to the 2025-12-11 read-only baseline:

Feature Default (read-only) Writable mode Hydration mode
Range GET reads ✅ ✅ (fallback) ✅ (during hydration)
LRU cache ✅ ✅ ✅
L0 polling ✅ ❌ disabled ✅
PRAGMA PITR ✅ undefined undefined
Writes ❌ ✅ via buffer ❌
Local file cache ❌ ❌ ✅ full DB

Applications without the Sprites-shaped cold-boot-speed requirement do not need either. "For ordinary read/write workloads, you don't need any of this mechanism. Litestream works fine without the VFS, with unmodified applications, just running as a sidecar alongside your application."

Last updated · 542 distilled / 1,571 read