CONCEPT Cited by 3 sources
Lossless Ethernet¶
Lossless Ethernet is the practice of running an Ethernet fabric that does not drop packets under congestion — the substrate RDMA-over-Converged-Ethernet (RoCE) historically assumes. Standard Ethernet drops when buffers fill; RoCE's reliable transport reacts badly to loss, so operators make the fabric lossless. The core mechanism is Priority Flow Control (PFC, IEEE 802.1Qbb): when a switch's ingress buffer for a traffic class fills past a threshold, it sends a PAUSE frame upstream to stop that class rather than drop. Congestion-control schemes like DCQCN (ECN + rate control) sit above PFC to throttle senders before PFC pauses are needed.
The costs of losslessness¶
PFC buys zero loss at real cost:
- Buffer headroom. A link needs enough buffer to absorb in-flight bytes between sending PAUSE and the sender stopping — headroom proportional to round-trip bytes = bandwidth × distance. The longer the cable, the more headroom.
- Head-of-line blocking and PFC spreading. A PAUSE stops a whole priority class, not one flow; pauses can propagate backward through the fabric, so congestion spreads beyond the link that caused it (PFC storms / congestion trees).
- Incast. Many senders → one receiver overwhelms buffers; losslessness turns the drop into back-pressure that must be managed, not avoided.
Two opposite responses on the wiki¶
Lossless Ethernet is the axis along which Meta's two generations of AI transport diverge:
- Lean into losslessness (2024 RoCE). Meta's SIGCOMM 2024 fabric ran PFC with deep-buffer spine switches and even turned DCQCN off at 400G for a year, relying on PFC + NCCL-level receiver-driven admission. The fabric is lossless; the endpoint trusts it. (Source: sources/2024-08-05-meta-a-roce-network-for-distributed-ai-training-at-scale)
- Reject losslessness (2026 MetaRoCE). MetaRoCE treats Ethernet as lossy by design — no PFC, no PAUSE frames — and moves repair to the endpoint NIC (per-path selective-ACK, per-packet spraying), precisely to escape PFC's head-of-line blocking and congestion-spreading. (Source: sources/2026-08-24-meta-metaroce-a-new-rdma-transport-built-for-ai-scale-ethernet)
Distance makes buffer headroom a first-order problem¶
At scale-across distances (tens of km across the BAG super-spine), the "headroom ∝ cable length" rule stops being a tuning detail. Provisioning every link for worst-case distance wastes buffer; getting it wrong lets congestion spread. Meta's answer: FBOSS measures actual cable length in place (timing probe traffic on an uncongested queue) and allocates PFC headroom against the measurement. (Source: sources/2026-10-12-meta-ocp-summit-2026-powering-ai-with-open-innovations-in-scale-up-scale-out-and-scale-across-networks)
Where it applies across domains¶
Meta reuses the same lossless RoCE class (routed IPv6, PFC, AI-tuned QoS) across scale-up (Tahan) and scale-out fabrics — losslessness is the unifying transport discipline over all three distance domains, even as the longest (scale-across) distances strain it hardest.
Related¶
- systems/roce-rdma-over-converged-ethernet — the transport that assumes losslessness.
- systems/metaroce — the clean-sheet transport that rejects losslessness.
- concepts/head-of-line-blocking — the failure mode PFC introduces.
- concepts/clos-topology — the fabric topology losslessness is layered over.
- systems/backend-aggregation-bag — where distance makes buffer headroom first-order.
- concepts/scale-up-scale-out-scale-across — the domains lossless RoCE spans.