Skip to content

PATTERN Cited by 1 source

In-loop decoder for real-time quality metrics

Intent

When computing reference visual-quality metrics (PSNR, SSIM, VMAF — see concepts/visual-quality-metric) for livestream transcoding where a post-hoc comparison is not possible, insert a decoder after each encoder in a multi-lane pipeline so the distorted frames are available for comparison against the pre-compression reference frames, all inside a single transcoding command.

Motivation

Reference metrics by definition need two streams:

  • Reference. The frames before compression (output of the shared source decoder).
  • Distorted. The frames after compression (what the encoder emitted).

For VOD the distorted stream is a finished file; one can run a separate comparison command afterward. For livestream there is no "afterward" — the encode is still emitting. Quality signals need to be computed during the encode or not at all.

"This is okay for offline or VOD use cases, but not for livestreaming where we might want to compute quality metrics in real time." (Source: sources/2026-03-09-meta-ffmpeg-at-meta-media-processing-at-scale)

Structure

                        ┌─> enc_A ─> dec_A ─┐
                        │                   ├─> PSNR/SSIM/VMAF
src ─> decoder ─────────┤                   ├─> PSNR/SSIM/VMAF
       (reference)      ├─> enc_B ─> dec_B ─┘
                        └─> enc_C ─> dec_C ─┘

The shared source decoder's output serves as the reference for each lane. After each encoder, an in-loop decoder runs to produce the distorted frames for that lane. A metric module compares reference and distorted per frame and emits per-lane scores continuously.

Per the post:

"To do this, we need to insert a video decoder after each video encoder used by each output lane. These provide bitmaps for each frame in the video after compression has been applied so that we can compare against the frames before compression. In the end, we can produce a quality metric for each encoded lane in real time using a single FFmpeg command line."

Upstream trajectory

FFmpeg did not support this graph shape natively. FFlabs, VideoLAN, and other contributors landed "in-loop" decoding in FFmpeg 7.0+, which was the final piece Meta needed to deprecate its internal FFmpeg fork for livestream use cases. Another instance of patterns/upstream-the-fix producing fork-retirement as the tangible architectural outcome.

Consequences

  • + Per-lane reference metrics are available live, during the encode.
  • + Enables quality-driven alerting and adaptive encoder-parameter tuning for livestreams.
  • + Works inside the same single-process pipeline as the deduplicated-decode pattern — no extra orchestration.
  • − Doubles the decode work per lane (one reference decoder, N distorted decoders).
  • − No-reference metrics do not need this machinery; using it when a no-reference metric suffices is waste.

When to use

  • Livestream transcoding where quality monitoring needs to be real-time.
  • Any pipeline producing quality telemetry alongside encoded outputs rather than as a separate post-hoc step.

When not to use

  • Offline / VOD quality analysis — cheaper to run the comparison as a separate post-encode FFmpeg command over the two finished files.
  • Production encode paths where the quality-metric cost (doubled decode work) is not justified by the operational signal.

Seen in

Last updated · 319 distilled / 1,201 read