Skip to content

CONCEPT Cited by 2 sources

LTX file format (LiteFS Transactions)

Definition

LTX — LiteFS Transactions — is the on-wire / on-disk file format Fly.io's LiteFS uses to represent sorted, per-transaction changesets of SQLite pages. Each LTX file is a sorted run of page-range records scoped to the time window its transactions cover. Reference implementation: superfly/ltx.

Because LTX files are sorted by page number and carry explicit transaction boundaries, they behave like an LSM-style sorted run over SQLite pages:

  • Multiple LTX files can be merged (k-way-merged by page number), producing a single LTX file that keeps only the latest version of each page over the combined time range.
  • Merging adjacent time windows is compaction; the resulting file is the point-in-time state of the database at the end of the window, represented as sorted pages.
  • A database can be reconstructed at any time t within retention by taking all LTX files whose window ends at or before t and compacting them.

From the 2025-05-20 Litestream redesign post:

"Each LTX file represents a sorted changeset of pages for a given period of time. Because they are sorted, we can easily merge multiple LTX files together and create a new LTX file with only the latest version of each page. … This is similar to how an LSM tree works." (Source: sources/2025-05-20-flyio-litestream-revamped)

Why this matters vs raw WAL shipping

The original Litestream design shipped raw WAL pages — every physical change to every page, regardless of whether the page was rewritten a thousand times in the retention window. Restore cost scaled with WAL volume, not with distinct logical pages touched. Restoring a database where one page was rewritten 1,000 times meant replaying 1,000 page-write records.

LTX changes the shape twice:

  1. Transaction-aware, not frame-aware. Each file is scoped to transactions (not WAL frames), so the unit of replay is a coherent commit, not a raw WAL fragment.
  2. Sortable and merge-compactable. Because LTX files are sorted, two LTX files covering consecutive windows k-way-merge into one file retaining only the latest version of each page — so restore to time t costs the size of the compacted state at t, not the cumulative WAL volume since the last snapshot.

That's why Ben Johnson calls out "minimal duplicate pages" as the key property post-compaction: in the compacted LTX, each page appears at most once.

Structural role in the LiteFS/Litestream stack

Before the 2025-05-20 Litestream redesign, LTX was LiteFS-only — LiteFS shipped LTX between primary and replica nodes over the network, while Litestream shipped raw WAL frames to object storage. The redesign converges the two tools on LTX:

  • LiteFS uses LTX for node-to-node replication (primary → replica).
  • Litestream (post-redesign) uses LTX for node-to-object-store replication (primary → S3 / Tigris).

This unification is the load-bearing design move behind the rest of the redesign: with LTX as the shared format, compaction, fast PITR, VFS-based read replicas, and wildcard-directory replication all follow.

Comparison vs SQLite WAL

Property SQLite WAL frame LTX file
Unit raw page write page-range in a committed transaction
Ordering append order sorted by page number
Merge-compactable no (would require sort) yes
Restore cost to time t proportional to frame count since snapshot proportional to compacted state at t
Multi-database support one WAL per DB one LTX stream per DB; sortable fleet-wide
Transaction awareness per-frame, boundary implicit explicit per-transaction

Visual

From the post:

linear-ltx

"A simple linear LTX file with 8 pages between 1 and 21."

merged-ltx

"Merging three LTX files into one."

2025-10-02 shipping disclosure: per-page compression + EOF index

The 2025-10-02 Litestream v0.5.0 shipping post (sources/2025-10-02-flyio-litestream-v050-is-here) adds two concrete implementation-level properties to the LTX format:

"It used to be an LTX file was just a sorted list of pages, all compressed together. Now we compress per-page, and keep an index at the end of the LTX file to pluck individual pages out. You're not seeing it yet, but we're excited about this change: we can operate page-granularly even dealing with large LTX files. This allows for more features. A good example: we can build features that query from any point in time, without downloading the whole database."

Two things changed:

  1. Whole-file compression → per-page compression. Pages are now individually compressed frames rather than a single compressed sorted run. Compression efficiency per byte drops slightly (compressor context smaller) but random access becomes cheap.
  2. End-of-file index. Each LTX file has an index at its tail mapping page-number → byte-offset, so a reader can HTTP range-GET a specific page without downloading the whole file.

Why this matters: this is the structural precondition for the VFS-based read-replica layer still on Litestream's roadmap. A read-replica needs to fetch just the pages a query touches, not the whole database, out of remote storage — which requires random-accessible pages inside LTX files.

Monotonic TXID as the addressing primitive

The 2025-10-02 post also names monotonic transaction IDs (TXIDs) as the LTX addressing primitive that retired Litestream's "generations":

"we have a monotonically incrementing transaction ID. We can use it to look up database state at any point in time, without searching across generations."

TXID is the addressing scheme LTX files use; the user-visible CLI now carries TXIDs where the pre-revamp generation/index/offset tuple used to appear (litestream wal → litestream ltx).

Seen in

  • sources/2025-05-20-flyio-litestream-revamped — canonical wiki disclosure; LTX framed as the shared file format that post-revamp unifies LiteFS + Litestream. Merge-and-compact visualization included.
  • sources/2025-10-02-flyio-litestream-v050-is-here — shipping-level disclosure. LTX library now compresses per-page (not whole-file) with an end-of-file index making individual pages random-accessible via HTTP range-GET — the structural precondition for VFS-based read replicas. Also confirms monotonic TXID as the addressing primitive replacing generation/index/offset (CLI: litestream wal → litestream ltx).
Last updated · 200 distilled / 1,178 read