Skip to content

CONCEPT Cited by 1 source

Multi-lane encoding pipeline

Definition

A multi-lane encoding pipeline produces multiple video encodings (lanes) from a single source in one coordinated pass. Each lane can differ in resolution, codec, framerate, or visual quality target, but all lanes share the same decoded source frames as input (Source: sources/2026-03-09-meta-ffmpeg-at-meta-media-processing-at-scale).

The shape is forced by DASH (and sibling protocols like HLS): clients select among pre-encoded renditions at runtime, so the server must produce the full ladder up front.

The naïve alternatives and their costs

One process per lane in series. Decode the source once per lane, encode. Wall-clock serial, compute-linear in the number of lanes.

One process per lane in parallel. Spawn N FFmpeg processes, each decoding the source and encoding its one lane. Wall-clock wins, but duplicates the decode N times and eats N × process-startup overhead. At Meta's scale — "over 1 billion video uploads daily, each requiring multiple FFmpeg executions" — this is unacceptable.

The efficient shape

One FFmpeg command, one decoder, N encoder instances. Decode the source once, feed the same frames to each encoder lane. This:

  • Deduplicates the decode.
  • Eliminates N-1 process startups.
  • Enables per-frame encoder parallelism when encoders co-exist in the same process — see next section.

See patterns/deduplicate-decode-across-encoder-lanes for the detailed pattern and its upstream trajectory.

Two layers of parallelism

  1. Intra-encoder threading. Each individual encoder is typically multi-threaded internally (motion estimation, rate control, entropy coding split across threads). This has been standard in FFmpeg for years.
  2. Inter-encoder (per-frame) threading. When multiple encoders attach to the same FFmpeg command, processing the current frame across all encoders in parallel rather than serially. Meta's internal FFmpeg fork had this as its headline differentiator; upstream FFmpeg 6.0 began the refactoring and FFmpeg 8.0 finished it — "the most complex refactoring of FFmpeg in decades".

Per-lane live quality metrics

A multi-lane pipeline is also the natural locus for in-loop decoding to compute per-lane VMAF/SSIM/PSNR during the encode — see concepts/in-loop-quality-metrics and patterns/in-loop-decoder-for-realtime-quality-metrics. Insert a decoder after each encoder in the graph; compare frames-after-compression against the shared frames-before-compression; emit per-lane metrics live.

Seen in

Last updated · 319 distilled / 1,201 read