Skip to content

SYSTEM Cited by 2 sources

JuiceFS

Definition

JuiceFS (juicefs.com, open-source on GitHub) is a POSIX-compatible distributed filesystem that splits storage into data chunks stored in an object-storage backend (S3-compatible, GCS, Azure Blob, MinIO, etc.) and metadata stored in a separate transactional database (Redis, MySQL, Postgres, SQLite, TiKV, etc.). A JuiceFS mount looks like a normal POSIX filesystem; reads and writes translate into metadata lookups + object-store GET/PUT operations, with local- disk chunk caching on the host.

Architectural shape

JuiceFS is the open-source reference implementation of the patterns/metadata-plus-chunk-storage-stack pattern:

  1. Metadata tier — a small transactional DB tracks inode tree, file attributes, and the map of file → chunks → object-store keys. Any of several DB backends is pluggable.
  2. Chunk tier — immutable, content-addressable chunks living in S3-shaped object storage. Byte volume lives here.
  3. Client cache — local-disk cache of hot chunks on the mounting host. Reads miss → GET from object store → populate cache.

The split lets operators pick transactional consistency (on the metadata DB) and durability/cost/availability (on the object store) independently, at the cost of running two systems.

Canonical wiki instance — Fly.io Sprites

Fly.io's Sprites disk stack is a fork of JuiceFS with the stock metadata backend replaced:

"The Sprite storage stack is organized around the JuiceFS model (in fact, we currently use a very hacked-up JuiceFS, with a rewritten SQLite metadata backend). It works by splitting storage into data ('chunks') and metadata (a map of where the 'chunks' are). Data chunks live on object stores; metadata lives in fast local storage. In our case, that metadata store is kept durable with Litestream. Nothing depends on local storage."

(Source: [[sources/2026-01-14-flyio-the-design- implementation-of-sprites]])

Fly.io's modifications (disclosed):

The post explicitly flags the stack as in flux: "I could easily do another 1500-2000 words here on the Cronenberg film Kurt came up with for the actual storage stack, but because it's in flux, let's keep it simple."

2026-02-04 sharpening: the metadata tier is Litestream VFS

sources/2026-02-04-flyio-litestream-writable-vfs discloses that the SQLite metadata backend runs on Litestream VFS — specifically in writable + hydration mode, making the metadata tier itself object-store-rooted (via LTX files + Range-GET page reads) with background-hydration to a local temp file for steady-state throughput. Johnson's framing of the Sprite-stack metadata store:

"The system that does this is JuiceFS, and the database — let's call it 'the block map' — is a rewritten metadata store, based (you guessed it) on BoltDB. I kid! It's Litestream SQLite, of course."

Worst-case block-map size: "low tens of megabytes." The cold-open latency budget is an HTTP-request one (the Sprite may be booting in response to an inbound request). This is a canonical instance of concepts/metadata-data-split-storage where the metadata tier recursively applies the same split-storage pattern — metadata rooted in object storage, served via a VFS that Range-GETs pages during cold open and hydrates to local file for steady-state reads.

Why a JuiceFS-lineage stack fits Sprites

The design requirement set Fly.io started from (100 GB durable per-Sprite storage, fast create, host-independent migration, fast checkpoint/restore) is almost exactly what JuiceFS's architecture optimises for:

  • Durability anchored at object storage — Fly.io's root-of-durability argument is the JuiceFS design premise.
  • Bus-hop-ish read latency — chunk cache on local NVMe keeps hot-path reads fast. Sprites' read-through cache extends this shape.
  • Host-independent state — the file system is "just" metadata + object-store keys; moving a workload between hosts is re-pointing metadata. Delivers concepts/durable-state-as-url.
  • Cheap snapshots — immutable chunks make metadata-clone snapshots O(metadata-size). Sprites lean on this for first-class checkpoints.
  • LSVD — Fly.io's earlier experiment putting Fly-Machine block devices behind object storage ("bottomless S3-backed volumes"). Ptacek's Sprites post: "We've always wanted to run Fly Machine disks off object storage [LSVD], but the performance isn't adequate for a hot Postgres node in production." JuiceFS+Sprites is the filesystem-level answer, not the block-device-level answer, for workloads that don't need Postgres-grade write latency.
  • Tigris — Fly.io's own regional S3-compatible object store. Likely (though not named) the substrate Sprites chunks land on. Tigris's architecture is itself a canonical instance of patterns/metadata-db-plus-object-cache-tier (FoundationDB metadata + NVMe byte cache + S3 origin). Sprites is a tenant of a metadata-plus-object-cache system running on top of another metadata-plus-object-cache system.
  • LiteFS — Fly.io's FUSE-based SQLite replication. Architecturally adjacent (FUSE + object-storage- for-durability) but different product shape — LiteFS is SQLite-specific; JuiceFS is general POSIX.

Caveats / open questions

  • JuiceFS stock metadata backend: the open-source project recommends Redis for production. Fly.io's SQLite+Litestream substitution is a significant deviation — its write-latency, crash-safety, and scale properties aren't discussed in the post.
  • Client-cache coherence: JuiceFS's open-source docs cover the metadata-DB-as-lock-server model for write coordination across mounts; Sprites are single-tenant (one VM per filesystem), so the usual multi-mount coherence discussion may not apply, but the post doesn't enumerate the assumption.
  • Compaction / garbage collection of chunks: JuiceFS performs background chunk GC driven by metadata; the Sprites post doesn't discuss cadence or lineage retention (which bears on checkpoint retention).
  • Amplification on cache miss (chunk size / prefetch policy / write alignment) undisclosed.

Seen in

  • [[sources/2026-01-14-flyio-the-design-implementation-of- sprites]] — canonical wiki instance. Fly.io's "very hacked-up JuiceFS, with a rewritten SQLite metadata backend" disclosure.
  • sources/2026-01-29-flyio-litestream-writable-vfs — metadata-backend disclosure sharpened: the SQLite tier runs on Litestream VFS in writable + hydration mode, making the metadata tier itself object-store-rooted (the "block map", "low tens of megabytes worst case"). Canonical wiki instance of JuiceFS's metadata slot being realized by a writable-VFS-over-object-storage SQLite — the metadata tier loses its "fast local storage" requirement and inherits the chunk tier's durability properties.
Last updated · 319 distilled / 1,201 read