Skip to content

CONCEPT Cited by 2 sources

LTX index trailer

Definition

The LTX index trailer is a small, fixed-format section at the tail of each LTX file containing an index that maps page number → byte offset (and size) inside the file. Disclosed as shipping in Litestream v0.5.0 (sources/2025-10-02-flyio-litestream-v050-is-here) alongside per-page compression; put to load-bearing use as the random-access primitive behind Litestream VFS in sources/2025-12-11-flyio-litestream-vfs.

Size and retrieval cost

From the 2025-12-11 post:

"LTX trailers include a small index tracking the offset of each page in the file. By fetching only these index trailers from the LTX files we're working with (each occupies about 1% of its LTX file), we can build a lookup table of every page in the database."

The ~1% of LTX file size datum is the key economic property: building a database-wide page index requires fetching ~1% of the total remote LTX bytes — for a 1 GB retention window, ~10 MB of index trailers.

Why it exists

The 2025-10-02 v0.5.0 post motivates the trailer as the structural precondition for random-access reads inside a large LTX file:

"It used to be an LTX file was just a sorted list of pages, all compressed together. Now we compress per-page, and keep an index at the end of the LTX file to pluck individual pages out. … we can operate page-granularly even dealing with large LTX files. This allows for more features. A good example: we can build features that query from any point in time, without downloading the whole database."

Two file-format changes ship together:

  1. Per-page compression (replacing whole-file compression) — each page is an independently-addressable compressed frame.
  2. EOF index trailer — maps page-number → byte-offset so a reader can locate any page without scanning the file.

Either change alone would be insufficient:

  • Per-page compression without the trailer would still require a full scan to locate a page by number.
  • An EOF trailer over whole-file-compressed content would give offsets, but the compressor's context window would span multiple pages and prevent true random decompress.

Composed with HTTP Range GET

The trailer's payoff comes from object-storage Range header support. A reader can:

  1. HTTP Range GET the last ~1% of each LTX file → index trailers.
  2. Build an in-memory (filename, page → byte-offset) lookup table.
  3. For each subsequent page request, HTTP Range GET exactly the page's bytes.

Canonical consumer: Litestream VFS (see patterns/vfs-range-get-from-object-store).

Seen in

  • sources/2025-10-02-flyio-litestream-v050-is-here — v0.5.0 disclosure of the trailer as a shipping file-format change, framed as "the structural precondition" for VFS-based read replicas.
  • sources/2025-12-11-flyio-litestream-vfs — canonical wiki-level instantiation. The ~1%-of-file-size datum is disclosed here; the trailer becomes the primary data structure Litestream VFS reads first on cold open to build its page-lookup table.
Last updated · 200 distilled / 1,178 read