Skip to content

CONCEPT Cited by 3 sources

SQLite Virtual Filesystem (VFS)

Definition

A SQLite Virtual Filesystem (VFS) is a plugin layer inside SQLite that abstracts the operations SQLite performs on the underlying OS — open, read, write, lock, sync, size, truncate, delete — into a set of function pointers an extension can override. See the upstream docs: sqlite.org/vfs.html.

Where SQLite normally writes to a real file via the OS VFS, a custom VFS can transparently substitute any storage backend without the application knowing — in-memory, networked, object storage, another SQLite instance, etc.

Why it's the FUSE alternative

Both LiteFS and (post-2025-05-20) Litestream need to intercept SQLite's I/O to do their job — LiteFS to capture transactions for replication, Litestream to serve pages from object storage in a read-replica. Two integration points exist:

  1. FUSE filesystem. LiteFS's primary approach — present a fake filesystem at the OS level; SQLite's normal file operations go through the FUSE layer, which captures them. "enough that users could use SQLite replicas just like any other database." But: "installing and running a whole filesystem (even a fake one) is a lot to ask of users." (Source: sources/2025-05-20-flyio-litestream-revamped)

  2. SQLite VFS. Load an extension into the application that overrides SQLite's storage layer directly — no FUSE, no kernel module, no root. LiteFS ships LiteVFS for this exact use case: "LiteFS can function without the FUSE filesystem if you load an extension into your application code, LiteVFS. LiteVFS is a SQLite Virtual Filesystem (VFS). It works in a variety of environments, including some where FUSE can't, like in-browser WASM builds."

The trade is surface area — FUSE intercepts every I/O transparently; VFS requires the application link the extension — but the deployment story is dramatically simpler.

Use in revamped Litestream

From the 2025-05-20 post:

"What we're doing next is taking the same trick and using it on Litestream. We're building a VFS-based read-replica layer. It will be able to fetch and cache pages directly from S3-compatible object storage."

Operationally: the application links the Litestream-VFS extension; SQLite reads go through the extension; pages come from a local cache, or on miss from Tigris or S3. No FUSE mount, no separate replica process, no local WAL handling on the replica side.

Caveat the post explicitly names: "this approach isn't as efficient as a local SQLite database. That kind of efficiency, where you don't even need to think about N+1 queries because there's no network round-trip, is part of the point of using SQLite."

2025-12-11: Litestream VFS ships

The proof-of-concept teased in 2025-05-20 and explicitly flagged as "not yet shipped" in the 2025-10-02 v0.5.0 post is now live as Litestream VFS (Source: sources/2025-12-11-flyio-litestream-vfs). Loadable via the standard SQLite extension mechanism:

sqlite> .load litestream.so
sqlite> .open file:///my.db?vfs=litestream

Concrete disclosures the shipping post adds to the VFS concept:

  1. Only the read side is overridden — "Litestream VFS handles only the read side of SQLite. Litestream itself, running as a normal Unix program, still handles the 'write' side." The existing write path is untouched.
  2. Page lookup via LTX EOF index trailer — "LTX trailers include a small index tracking the offset of each page in the file. By fetching only these index trailers from the LTX files we're working with (each occupies about 1% of its LTX file), we can build a lookup table of every page in the database." See concepts/ltx-index-trailer.
  3. Range GET against S3-compatible storage — page reads are resolved by HTTP byte-range GETs against the object store. Canonical instance of patterns/vfs-range-get-from-object-store.
  4. LRU cache of hot pages — "Most databases have a small set of 'hot' pages — inner branch pages or the leftmost leaf pages for tables with an auto-incrementing ID field." The 2025-05-20 "isn't as efficient as a local SQLite database" caveat is partially mitigated via the LRU cache.
  5. SQL-level PITR — PRAGMA litestream_time = '5 minutes ago'; redirects reads in the current session to the database state at the chosen timestamp. See concepts/pragma-based-pitr.

2026-02-04: Writable VFS and hydration

sources/2026-02-04-flyio-litestream-writable-vfs extends Litestream VFS past its original read-only shape — the VFS interface is now intercepted on the write side too, and a second env-var-gated mode adds a background-populate-local-file loop that serves as a DB-level sibling of dm-clone-style hydration. Both are explicitly narrow-fit features for Fly Sprites' storage stack, not new defaults.

  1. Write interception — with LITESTREAM_WRITE_ENABLED=true, SQLite writes through the VFS go to a local buffer that syncs to object storage every ~1 s (or on clean shutdown). Single-writer only; polling for remote writers is disabled. "Multiple-writer distributed SQLite databases are the Lament Configuration." The VFS surface now covers both I/O directions, not just read.
  2. Hydration as a VFS mode — with LITESTREAM_HYDRATION_PATH=/tmp/db.sqlite, the VFS serves queries from object storage while a background loop pulls the whole database (via LTX compaction) to a local temp file; once hydration completes the read path switches over to the local file. "We shoplifted a trick from systems like dm-clone." Canonical VFS-layer instance of concepts/async-clone-hydration applied at SQLite-database granularity (rather than block-level or git-repo-level).
  3. Use case: the Sprite "block map." Fly.io's JuiceFS fork runs its metadata tier on SQLite + Litestream VFS in this combined mode — the metadata tier is itself object-store-rooted, and the HTTP-request- timely boot budget is the gating latency constraint.

The shipping disclosure ties the VFS back to the general principle: the SQLite VFS interface is a powerful interception point not just for changing where pages come from but for changing how the whole I/O lifecycle is managed, at essentially-application-code granularity (single loadable extension, no kernel module, no FUSE mount, no new process).

Environments where VFS is the only option

  • In-browser WASM SQLite builds — no FUSE available in the browser; VFS is the only surface.
  • Restricted FaaS / PaaS environments — no kernel module loading, no privileged filesystem mount, but user code can link an extension.
  • Platforms with tight sandboxing — same reasoning: you control the process, not the kernel.

Seen in

  • sources/2025-05-20-flyio-litestream-revamped — canonical wiki introduction; VFS as the replacement for FUSE in the Litestream read-replica design; LiteVFS as the precedent from LiteFS.
  • sources/2025-12-11-flyio-litestream-vfs — shipping disclosure. Litestream VFS loads as a standard SQLite extension (.load litestream.so), overrides only the read side, resolves page reads via HTTP Range GETs against LTX files in object storage using the ~1%-sized EOF index trailer per file, fronted by an LRU cache of SQLite's hot B-tree pages. Adds SQL-level PITR (PRAGMA litestream_time) and L0-polling-based near-realtime replica behaviour. Canonical wiki instance of VFS-range-GET-from-object-store composition.
  • sources/2026-02-04-flyio-litestream-writable-vfs — VFS interface extended to writes + a hydration mode. LITESTREAM_WRITE_ENABLED=true intercepts writes (single- writer, ~1 s buffered sync); LITESTREAM_HYDRATION_PATH populates a local temp file in the background while remote reads continue. First wiki instance of the SQLite VFS interface intercepting the whole I/O lifecycle — not just the read path — for an object-store-rooted setup.
Last updated · 542 distilled / 1,571 read