Skip to content

Fly.io — Litestream VFS

Summary

Ben Johnson's shipping post for Litestream VFS — the SQLite VFS extension teased as proof-of-concept in the [[sources/2025-05-20-flyio-litestream-revamped|2025-05-20 revamp]] and flagged as "still not shipped" in the [[sources/2025-10-02-flyio-litestream-v050-is-here|2025-10-02 v0.5.0 post]] is now live. Load the shared library (.load litestream.so), open a database with file:///my.db?vfs=litestream, and SQLite runs queries "hot off an object storage URL" — no full database download, no local replica process, no FUSE, no modification to the SQLite library the application already links. Individual page reads are satisfied by HTTP Range GETs against the LTX files in S3-compatible storage, using a new end-of-file index trailer (~1% of each LTX file) to map page-number → byte-offset. An LRU cache fronts the range reads, exploiting the fact that most databases have a small hot set ("inner branch pages or the leftmost leaf pages for tables with an auto-incrementing ID field"). Because LTX compaction runs continuously and the VFS can poll the L0 level (uploads every second, retained until compacted to L1), the result is "a near-realtime replica" that also "starts up really fast" — useful for "the AIs and the agents and the clouds and the hoyvin-glavins" ephemeral-server era. Point-in-time recovery is now expressed in SQL via a new PRAGMA litestream_time — set it to a relative ('5 minutes ago') or absolute ('2000-01-01T00:00:00Z') timestamp and subsequent queries read from that point in backups. "Instantaneous point-in-time recovery (PITR), expressed simply in SQL and SQLite pragmas." Litestream itself (the Unix program) still handles the write side; the VFS extension handles only the read side — SQLite's application-transparent Litestream story is unchanged for the write path.

Key takeaways

  1. Litestream VFS ships as a SQLite extension (.load litestream.so), activated via the SQLite URI filename vfs parameter: file:///my.db?vfs=litestream. It is "a plugin for the SQLite you're already using" — not a new version of SQLite, not a replacement library. Standard SQLite loadable- extension mechanism; works anywhere SQLite does (browser WASM, restricted FaaS, etc.), exactly the environments the VFS was pitched for in the 2025-05-20 design post. (Source: sources/2025-12-11-flyio-litestream-vfs)

  2. The VFS overrides only the read side of SQLite's file interface. "We override only the few methods we care about. Litestream VFS handles only the read side of SQLite. Litestream itself, running as a normal Unix program, still handles the 'write' side." SQLite's Read() call carries a byte offset computed against the "local file" illusion; the VFS translates that to "the remote filename, the 'real' byte offset into that file, and the size of the page" by looking up in the page index it has built from LTX trailers.

  3. Page-level reads via HTTP Range GET against object storage. "That's enough for us to use the S3 API's Range header handling to download exactly the block we want." No whole-database download at open time; no replica process; no local WAL handling on the read side. See patterns/vfs-range-get-from-object-store.

  4. LTX end-of-file index trailers enable random access. "LTX trailers include a small index tracking the offset of each page in the file. By fetching only these index trailers from the LTX files we're working with (each occupies about 1% of its LTX file), we can build a lookup table of every page in the database." The LTX file format's per-page compression + EOF index (disclosed as shipped in the v0.5.0 post) is what makes this cheap — fetch ~1% of each relevant LTX file to build a database-wide page index. See concepts/ltx-index-trailer.

  5. LRU cache of hot pages inside the VFS. "To save lots of S3 calls, Litestream VFS implements an LRU cache. Most databases have a small set of 'hot' pages — inner branch pages or the leftmost leaf pages for tables with an auto-incrementing ID field. So only a small percentage of the database is updated and queried regularly." Exploits the specific B-tree hot-page shape of SQLite (inner branches + leftmost leaves) that the 2025-10-02 "sandwiches" worked example already canonicalised as the AUTOINCREMENT-primary-key reality.

  6. Near-realtime replica via L0 polling + incremental index update. "Because Litestream backs up (into the L0 layer) once per second, the VFS code can simply poll the S3 path, and then incrementally update its index. The result is a near-realtime replica. Better still, you don't need to stream the whole database back to your machine before you use it." See patterns/near-realtime-replica-via-l0-polling.

  7. PITR as a SQL-level knob via PRAGMA litestream_time. PRAGMA litestream_time = '5 minutes ago'; (relative) or '2000-01-01T00:00:00Z' (absolute) sets the query-time backup-read point. "What we're doing here is instantaneous point-in-time recovery (PITR), expressed simply in SQL and SQLite pragmas." Replaces the CLI-level litestream restore -timestamp flow for exploratory queries. See concepts/pragma-based-pitr.

  8. Compaction ladder mechanics now explicit at the L0 level. "By default, Litestream uses time intervals of 1 hour at the highest level, down to 30 seconds at level 1. L0 is a special level where files are uploaded every second, but are only retained until being compacted to L1." The LTX compaction ladder gets a new L0 disclosure on top of the 30s / 5m / 1h L1-L2-L3 numbers from the v0.5.0 post — the seconds-granularity polling required for a near-realtime replica lives at L0, which is ephemeral (not retained past L1 compaction). Full snapshots live above L3 on a daily cadence.

  9. Fast startup — "living in an age of increasingly ephemeral servers." "All this smoke-and-mirrors of querying databases without fully fetching them has another benefit: it starts up really fast! We're living an age of increasingly ephemeral servers, what with the AIs and the agents and the clouds and the hoyvin-glavins. Wherever you find yourself, if your database is backed up to object storage with Litestream, you're always in a place where you can quickly issue a query." Positions the VFS as the primitive an agentic coding platform (Phoenix.new-style) actually wants — the "agent-PITR" thesis from the 2025-05-20 post gets its shipped artefact.

  10. PITR restores across the full retention window, single- second resolution. "Litestream holds backup files for every state your database has been in, with single-second resolution, for as long as you want it to. Forgot the WHERE clause on a DELETE statement? Updating your database state to where it was an hour (or day, or week) ago is just a matter of adjusting the LTX indices Litestream manages." PITR is now an index adjustment, not a restore job — the worked UPDATE sandwich_ratings SET stars = 1 DROP-WHERE example in the post demonstrates rewinding a live prod disaster on a dev machine in two pragma lines.

  11. "You could write Litestream yourself" — the explicit design posture. "Litestream is solid for serious production use (we rely on it for important chunks of our own Fly.io APIs). But you could write Litestream yourself, just from the basic ideas in these blog posts. We think that's a point in its favor. We land there because the heavy lifting in Litestream is being done by SQLite itself, which is how it should be." Continues Fly.io's self-described "keep the underlying mechanism simple enough that you can fit your head around it" framing from the 2025-05-20 revamp post.

  12. Opt-in, additive, doesn't replace the rest of Litestream. "you don't have to use our VFS library to use Litestream, or to get the other benefits of the new LTX code." Same point made in the 2025-10-02 post: v0.5.0's LTX + compaction + CASAAS + NATS JetStream replica are all usable without ever loading the VFS extension. The VFS is a read-side performance + PITR primitive, not a required new mode.

Systems extended

  • Litestream — the VFS read-replica layer is now shipped. The 2025-10-02 v0.5.0 post explicitly flagged the VFS as "proof of concept working, not yet shipped"; this post is its ship announcement. SQL-level PRAGMA litestream_time is a new user surface. [extends]
  • Litestream VFS — new system, the SQLite loadable-extension realising the VFS read-replica layer. Overrides only Read() + related methods on SQLite's I/O interface; satisfies reads from an LRU cache + HTTP range GETs against LTX files in S3; polls L0 for near-realtime replica behaviour; exposes PRAGMA litestream_time for SQL-level PITR.
  • SQLite — this post reinforces SQLite's VFS extension surface as the integration point that lets an unmodified SQLite library transparently read pages from S3. "In particular: Litestream VFS doesn't replace the SQLite library you're already using. It's not a new 'version' of SQLite. It's just a plugin for the SQLite you're already using." [extends]
  • LiteFS — the architectural-influence entry gains a shipped-artefact counterpart: LiteFS's LiteVFS was the precedent, and now Litestream has its own VFS shipped. Convergence is complete — LTX as shared wire format + VFS as shared integration surface. [extends]
  • S3 — the Range header handling is explicitly named as the primitive Litestream VFS leans on: "modern object storage providers all let us fetch slices of files". S3's byte-range-GET semantic is the underlying capability that makes page-level reads from remote LTX files tractable. [extends]

Concepts extended

  • SQLite Virtual Filesystem (VFS) — the previously-theoretical "Litestream VFS read-replica layer" is now an actual shipped VFS extension named here. The caveat the 2025-05-20 post flagged ("this approach isn't as efficient as a local SQLite database") gets its first real-deployment mitigation: the LRU cache of hot B-tree pages. [extends]
  • LTX file format — end-of-file index trailer is disclosed as ~1% of each LTX file; fetching only the trailers from relevant LTX files is enough to build a database-wide page index. The per-page compression + EOF index shipped in v0.5.0 (2025-10-02) is what this post makes useful. [extends]
  • LTX index trailer — new concept. The page-index section at the tail of each LTX file; maps page number → byte offset within the file. Single-HTTP-range-read per LTX file is enough to hydrate the full page-lookup table for the window that LTX file covers.
  • PRAGMA-based PITR — new concept. A SQL-level knob (PRAGMA litestream_time = '<when>';) that repositions subsequent query reads at a specific historical timestamp. Converts PITR from an out-of-band CLI operation into an in-query parameter — callers can mix historical and live reads in a single session.
  • Point-in-time recovery — this post's shipping artefact ("instantaneous PITR, expressed simply in SQL and SQLite pragmas") is the first shipped SQL-native PITR surface in this wiki's SQLite/Litestream coverage. The two-pragma worked example (PRAGMA litestream_time = '5 minutes ago'; SELECT * …) replaces the pre-shipping litestream restore -timestamp CLI flow for ad-hoc dev/debug reads. [extends]
  • LRU cache — mentioned here as the VFS's S3-call-reduction mechanism; not promoted to a standalone concept page because LRU is ubiquitous. The SQLite-specific observation ("inner branch pages or the leftmost leaf pages for tables with an auto-incrementing ID field") is recorded inline on systems/litestream-vfs as the hot-set shape the cache exploits.

Patterns extended

  • VFS-driven Range GET from object storage — new pattern. Application-linked VFS/extension translates a page-request from a SQLite-library caller into an HTTP byte-range GET against a remote LTX file, using an EOF index trailer (fetched once per relevant LTX file) to map page → offset. LRU-cached on the hot-set. Canonical instance: Litestream VFS.
  • Near-realtime replica via L0 polling — new pattern. A read-replica of an object-store-backed database keeps the local page index approximately current by polling the uploader's finest-grained level (L0 in the Litestream case, 1-second upload cadence) and incrementally updating its index. No streamed-WAL replication needed, no local write journal. Canonical instance: Litestream VFS.
  • LTX compaction — the compaction ladder gains an explicit L0 disclosure on top of the L1/L2/L3 (30s/5m/1h) shape from the v0.5.0 post. L0 is a "special level" that accepts one LTX file per second but retains them only until L1 compaction — the seconds-grained polling surface the VFS uses to stay near-realtime. Also confirms daily full snapshots above L3. [extends]
  • SQLite + LiteFS + Litestream — the Litestream leg now covers the read side too, via the VFS extension. The three-layer pattern that was *"LiteFS = availability + read-scaling; Litestream = durability
  • disaster recovery" gains a fourth shape: Litestream (+ VFS) = durability + remote-read + SQL-PITR*. Litestream VFS doesn't replace LiteFS for low-latency cross-region replication, but it does give a FUSE-free read-replica primitive in environments where FUSE is impossible or LiteFS is overkill. [extends]

Operational numbers

  • L0 upload cadence: 1 file per second; retained only until L1 compaction ("a special level").
  • L1 / L2 / L3 windows: 30-second / 5-minute / 1-hour (from v0.5.0 post; confirmed here as "By default, Litestream uses time intervals of 1 hour at the highest level, down to 30 seconds at level 1").
  • Daily full snapshots above L3 ("In the diagram below, we're taking daily full snapshots").
  • LTX EOF index trailer size: ~1% of each LTX file.
  • Restore granularity: single-second resolution across the full retention window ("Litestream holds backup files for every state your database has been in, with single-second resolution, for as long as you want it to").
  • PITR at query time: zero-RTO ("instantaneous" — index adjustment, not data movement); cost is the HTTP range reads the query actually touches.
  • No latency numbers disclosed: cold-open time against an unknown database, warm-query latency with populated LRU, range- GET p99 to S3/Tigris, LRU hit-rate under representative SQLite workloads — all undisclosed.
  • No S3-cost numbers disclosed: cost per VFS-query-session, LRU size needed to hit a target S3-request-per-query budget, egress cost per session — undisclosed.

Caveats

  • Less efficient than a local SQLite database. Re-stated from the 2025-05-20 design post: "this approach isn't as efficient as a local SQLite database". The LRU cache softens this but doesn't eliminate it — any cache miss is an HTTP round-trip, and SQLite workloads that rely on "no network round-trip, so N+1 queries don't matter" semantics can be pathological against a remote VFS.
  • Single-writer SQLite ceiling unchanged. Writes still go through the regular Litestream Unix program on the primary; the VFS is read-only. The three-layer stack's single-writer assumption carries forward.
  • No SLOs for L0-polling freshness. Post asserts "near-realtime" but does not define the read-your-writes lag ceiling against the primary's L0 upload cadence. 1-second upload cadence + polling interval on the replica + L0→L1 compaction timing all compose into an effective read-lag that isn't quantified.
  • LRU cache sizing undisclosed. The litestream.so extension's default cache size, eviction policy details, and shared-vs-per- connection cache scope aren't specified; the post leans on SQLite B-tree hot-set properties without quantifying cache hit rates.
  • No LRU cache invalidation story on L0 poll. Post describes incremental index update on L0 refresh but doesn't detail how cached pages that got superseded by a newer LTX frame are invalidated; presumably index refresh + LRU key-change handles it, but the mechanics aren't spelled out.
  • PITR + write coexistence undefined. PRAGMA litestream_time positions reads in the past; the post is silent on what happens if the same session tries a write with a non-zero litestream_time set. Presumably writes still go to the normal Litestream primary (the VFS is read-side-only), but the semantic — do you get a write-to-present-despite-reading-past? an error? — isn't specified.
  • Tigris / GCS / Azure parity not claimed. Post names S3-compatible object storage generically and shows the AWS Range header doc; doesn't enumerate per-provider behaviour for byte- range GET cost, throughput ceilings, or range-caching behaviour that would affect Litestream VFS economics in practice.
  • No concurrent-VFS-reader disclosure. Post demonstrates one SQLite shell opening the database; doesn't cover whether multiple processes on the same host can share an LRU cache, whether distinct hosts running the VFS against the same path will duplicate range GETs, or whether any coordination exists.
  • No shipping-version stamp. Post does not cite a Litestream version tag (contrast the 2025-10-02 v0.5.0 post's explicit version label); presumably the VFS is shipping in a post-v0.5.0 release, but the specific version is not stamped.
  • No production-deployment numbers. The 2025-10-02 post claimed Fly.io runs Litestream for "important chunks of our own Fly.io APIs". This post repeats the claim but does not disclose how many databases, what access pattern, or what typical workload shape — no self-reported deployment telemetry.

Source

Last updated · 200 distilled / 1,178 read