Skip to content

CONCEPT Cited by 2 sources

Shadow WAL (Litestream legacy mechanism)

Definition

The shadow WAL is the mechanism the original 2020 Litestream used to get a coherent, durable copy of a SQLite WAL out of a live database without modifying the application or SQLite itself. It is named explicitly in Ben Johnson's 2025-05-20 redesign post as the legacy mechanism being retired in favour of the LTX file format.

From the post:

"Here's how Litestream was originally designed: you run litestream against a SQLite database, and it opens up a long-lived read transaction. This transaction arrests SQLite WAL checkpointing, the process by which SQLite consolidates the WAL back into the main database file. Litestream builds a 'shadow WAL' that records WAL pages, and copies them to S3." (Source: sources/2025-05-20-flyio-litestream-revamped)

How it worked

Three mechanisms composed:

  1. Long-lived read transaction — SQLite's MVCC semantics guarantee a reader sees a consistent snapshot; as long as the Litestream reader's transaction is open, SQLite will not checkpoint (consolidate the WAL into the main file) past the reader's point.
  2. Shadow WAL — Litestream reads the live WAL frames in order (they're not going anywhere, because checkpointing is arrested) and copies them to a staging area on disk, which it then uploads to object storage.
  3. Restore by replay — to restore, Litestream downloads snapshot(s) + all WAL frames since, and replays them into a fresh database.

The elegance: the application sees a normal SQLite database. Litestream is an external process that reads the WAL without Intercepting or modifying anything in the write path.

Why it was retired

The 2025-05-20 post names the key shortcoming:

"This is simple, which is good. But it can also be slow. When you want to restore a database, you have have to pull down and replay every change since the last snapshot. If you changed a single database page a thousand times, you replay a thousand changes. For databases with frequent writes, this isn't a good approach."

Root cause: shadow WAL shipping preserves raw WAL frames — every physical page-write, regardless of whether the same page was rewritten 999 times afterwards. Restore cost therefore scales with WAL volume since the last snapshot, not with distinct logical state at the restore target.

The replacement: LTX + compaction

The revamp imports LiteFS's LTX file format — sorted, transaction-aware page-range changesets. Because LTX files are sortable, they can be merge-compacted into a single file that holds only the latest version of each page over the combined window (see patterns/ltx-compaction). Restore to any PITR target now costs the compacted state size, not the cumulative write history.

Secondary consequence: wildcard replication

Shadow-WAL polling was per-database and expensive; it made directory/wildcard replication (e.g. /data/*.db with hundreds of databases) infeasible. LTX, by removing the polling-per-db cost, removes that ceiling:

"In the old Litestream design, WAL-change polling and slow restores made it infeasible to replicate large numbers of databases from a single process. … Now that we've switched to LTX, this isn't a problem any more."

Retirement confirmed in v0.5.0 (2025-10-02)

The 2025-10-02 Litestream v0.5.0 shipping post (sources/2025-10-02-flyio-litestream-v050-is-here) confirms the retirement as shipped:

"Due to the file format changes, the new version of Litestream can't restore from old v0.3.x WAL segment files. That's OK though! The upgrade process is simple: just start using the new version. It'll leave your old WAL files intact, in case you ever need to revert to the older version."

Shadow-WAL segment files are no longer readable by Litestream v0.5.0+ — the migration is a cutover, not a cross-format restore. v0.5.0 leaves old WAL files on disk so v0.3.x remains a rollback option, but there is no bridge that can serve PITR spanning the format boundary.

The CLI surface also migrated: litestream wal is renamed to litestream ltx, and all user-visible references to generation/index/offset become monotonic transaction IDs (TXIDs). The shadow-WAL mechanism is now historical on the wiki, not current-pre-retirement.

Seen in

Last updated · 200 distilled / 1,178 read