Skip to content

CONCEPT Cited by 1 source

Bytes in flight

Definition

Bytes in flight (abbreviated bytes_in_flight, sometimes bif) is the number of bytes the sender has transmitted but not yet acknowledged at any moment. It is the canonical transport-layer state variable that every loss-based CCA (and every pacing-based CCA) tracks explicitly, because the fundamental rate-limit rule is:

bytes_in_flight ≤ cwnd at all times.

The sender may transmit new data only if doing so would keep bytes_in_flight ≤ the cwnd. When an ACK arrives, bytes_in_flight decreases by the acknowledged byte count, freeing up headroom for the next send.

Why the distinction between "congestion-limited" and

"application-idle" matters

The 2026-05-12 Cloudflare quiche post (Source: sources/2026-05-12-cloudflare-when-idle-isnt-idle-how-a-linux-kernel-optimization-became-a-quic-bug) is a canonical case study in why bytes_in_flight == 0 is ambiguous as a signal:

  • Interpretation A — "the application is idle." The peer has no data to send right now. The connection is in a true idle period; any CCA growth-curve time-tracking should be paused or shifted forward.
  • Interpretation B — "the pipe drained because cwnd is tiny." At minimum cwnd (two packets), every ACK cycle drains the pipe to zero between the last ACK and the next send. The application had data ready but was congestion-limited, not idle. Treating this as Case A is what triggers the CUBIC minimum-cwnd death spiral.

The 2020 quiche port of the Linux-kernel 2017 CUBIC-after-idle fix used bytes_in_flight == 0 as the idle predicate — which is correct at large cwnd (the distinction rarely matters because bytes_in_flight doesn't reach zero during active transfers) but wrong at minimum cwnd, where the distinction is load-bearing on every ACK cycle.

The fix: distinguish Case A from Case B by measurement

Cloudflare's 2026-05-12 fix added a last_ack_time state variable and uses max(last_ack_time, last_sent_time) as the idle-delta anchor:

  • Case A (true idle). No ACKs in a long time → last_ack_time is far in the past → the computed delta captures the genuine idle duration → CUBIC's epoch is correctly shifted forward.
  • Case B (congestion-limited, transient zero). Last ACK landed microseconds ago → last_ack_time ≈ now → the delta is near zero → no spurious epoch advance → no death spiral.

See patterns/measure-idle-from-last-ack-not-last-send for the general pattern.

Practical consequences

  • At large cwnd, bytes_in_flight oscillates well above zero. The Case-A/B distinction doesn't bite — most CCAs' idle-detection heuristics work correctly.
  • At minimum cwnd (typically 2 × MSS ≈ 2,700 bytes), every round-trip drains bytes_in_flight to zero between the last ACK and the next send. Any logic that treats this as idleness and performs time-forward adjustments can lock the connection into a death spiral.
  • The ACK clock is the forcing function. Because ACKs are RTT-periodic and (on downloads) the server sends the next burst in response to ACKs, the Case-B scenario repeats once per RTT — which is why the bug's oscillation period matches the RTT (see concepts/ack-clock).

Seen in

Last updated · 542 distilled / 1,571 read