Skip to content

SYSTEM Cited by 1 source

Live Coder

Definition

Live Coder is Dropbox's on-the-fly erasure-coding service inside Magic Pocket. Its original role: write data directly into erasure-coded volumes, bypassing the previous replicated-then-re-encoded background path, which reduced write amplification for background writes.

Rolled out gradually over several months across regions.

Original design target

  • Fewer backend storage operations per logical write.
  • Not intended for latency-sensitive traffic — well-suited to background-throughput workloads.
  • EC-native: produces erasure-coded volumes without the replicated intermediate.

Unintended side effect

Volumes created through the Live Coder path were severely under-filled — in the worst cases, <5% of allocated capacity held live data. Each under-filled volume consumed a full disk allocation (Magic Pocket volumes are fixed-size, immutable once closed), so storage overhead rose sharply across the fleet as the rollout progressed. The existing baseline L1 compaction strategy could not reclaim the long tail of sparse volumes fast enough, forcing the multi-strategy compaction redesign (L1 + L2 + L3).

Second role — as reclamation engine

In the L3 compaction strategy (patterns/streaming-re-encoding-reclamation), the Live Coder is repurposed as a streaming reclamation pipeline:

  • Live blobs drained from severely-under-filled source volumes stream into the encoder.
  • The encoder accumulates and emits new, durable erasure-coded volumes over time.
  • Each source volume is reclaimed immediately once drained.

Canonical statement from the post: "Compaction is, in effect, a constrained form of re-encoding." Live Coder provides the re-encoder; L3 provides the drain + accumulate + reclaim loop on top of it.

Seen in

Last updated · 200 distilled / 1,178 read