Skip to content

SYSTEM Cited by 3 sources

LiteFS

LiteFS (fly.io/docs/litefs) is Fly.io's primary/replica distributed SQLite system. It presents as a FUSE filesystem that intercepts writes to a SQLite database file on the primary node and ships those changes to replica nodes, so that unmodified SQLite libraries on any node can query a locally-mounted, replicated copy of the database.

What it does for Fly's tkdb

LiteFS is the replication substrate underneath tkdb, Fly.io's Macaroon token authority:

"tkdb is about 5000 lines of Go code that manages a SQLite database that is in turn managed by LiteFS and Litestream. It runs on isolated hardware (in the US, Europe, and Australia) … LiteFS gives us subsecond replication from our US primary to EU and AU, allows us to shift the primary to a different region, and gives us point-in-time recovery of the database." (Source: sources/2025-03-27-flyio-operationalizing-macaroons.)

Load-bearing properties in that deployment:

  1. Subsecond cross-region replication — US primary → EU + AU replicas. Read-after-write lag is tolerable for Macaroon verification because most verifications hit a client-side cache (patterns/verification-cache-with-revocation-feed).
  2. Primary failover — the primary can be shifted to another region if the US region becomes unreachable. Critical for a single-writer service that's on the authenticate-everything hot path.
  3. Unmodified SQLite clients — the tkdb Go code uses the stock mattn/go-sqlite3 (or equivalent) driver and knows nothing about LiteFS. Replication is a filesystem concern.

Design shape (from Fly's own docs)

  • Single writer, many readers. SQLite itself enforces single-writer semantics, and LiteFS extends the invariant across the cluster — only the primary node accepts writes; replicas forward writes by not accepting them (the application is expected to route writes to the primary).
  • LTX (LiteFS Transactions) — LiteFS's on-the-wire log format. Each committed SQLite transaction emits an LTX record shipped to replicas.
  • Consul / FlyConsul for primary election — LiteFS uses a Consul lock to pick the primary; on failure, another node claims the lock and becomes primary.

Pairing with Litestream

LiteFS handles node-to-node replication (subsecond, hot). Litestream handles PITR to object storage (durable, cold). The combination is the patterns/sqlite-plus-litefs-plus-litestream stack that Fly ships internally and recommends externally:

  • LiteFS = availability + read-scaling.
  • Litestream = durability + disaster recovery.
  • SQLite = on-disk format + query surface.

Seen in

  • sources/2025-03-27-flyio-operationalizing-macaroons — canonical wiki instance; LiteFS replicates tkdb's token DB across US/EU/AU with subsecond cross-region lag and primary-failover support.
  • sources/2025-05-20-flyio-litestream-revamped — architectural-influence entry. LiteFS is the source of every idea the 2025-05-20 Litestream revamp imports: (1) the LTX file format — originally LiteFS's on-wire log format for node-to-node replication, now shared with Litestream for node-to-object-store replication; (2) LiteVFS — LiteFS's SQLite VFS extension (for environments where FUSE isn't available, like WASM) — is the precedent for Litestream's new VFS-based read-replica layer. The post also names LiteFS's Consul dependency for primary election as the usability wall "probably part of the reason Litestream is so much more popular than LiteFS." Litestream's CASAAS (conditional-write lease) avoids the Consul dependency by reusing the object store.
  • sources/2025-10-02-flyio-litestream-v050-is-here — convergence-shipped entry. Litestream v0.5.0 ships with LTX as the on-wire / on-disk format — the same format LiteFS uses for node-to-node replication. The architectural convergence foreshadowed in the 2025-05-20 design post is now concrete in the shipping artifact: both tools read and write LTX. The v0.5.0 upgrade also adds per-page compression + end-of-file index to the LTX library, giving page-granular random access — the precondition for VFS-based read replicas (Litestream's read-replica layer is still proof-of-concept, not shipped in v0.5.0).
Last updated · 200 distilled / 1,178 read