Skip to content

SYSTEM Cited by 3 sources

Backend Aggregation (BAG) super-spine

Backend aggregation (BAG) is Meta's centralized, Ethernet-based super-spine that interconnects many spine-layer training fabrics into one mega-cluster spanning buildings, campuses, and regions — the scale-across layer of Meta's AI networking stack. It is what lets a single logical cluster outgrow any single region. BAG was first detailed in Meta's Feb-2026 "Building Prometheus" post and reprised at OCP Summit 2026.

Scale

  • Operates at petabit scale; inter-BAG capacity between a pair of regions reaches tens of petabits per second.
  • Underpins Prometheus (~1 GW in 2026) — functioning as a single cluster even though its capacity is spread across conventional data-center buildings, weatherproof outdoor structures, and adjacent colocation space — and scales toward Hyperion (~5 GW).
  • Aggregates both DSF and NSF (and FPF) building-scale fabrics across sub-regions into a unified training network.

Distance becomes a first-order design constraint

At BAG reach, distance itself is a design constraint, in three ways:

  1. Buffer headroom scales with cable length. Lossless Ethernet requires buffer headroom proportional to cable length; once links stretch tens of kilometers, buffer management becomes a first-order problem, not a tuning detail — and getting it wrong risks congestion spreading beyond the link that caused it. Instead of provisioning every link for worst-case distance, FBOSS now measures actual cable length in place — timing probe traffic on an uncongested queue to derive round-trip distance — and allocates headroom against the measurement.

  2. Load balancing over degraded (not just down) paths. BAG uses eBGP link-bandwidth signaling (see concepts/bgp) so traffic is distributed in proportion to the capacity actually available on each path, with deliberate damping so a single link failure doesn't ripple routing churn across the domain.

  3. Collectives can fall off a latency cliff. Because inter-fabric latency is several times intra-fabric latency, collective-communication libraries — which generally pick algorithms without awareness of network distance — can degrade sharply. Meta attacks this from both ends: adaptive-routing improvements to cut the latency penalty, and scheduling that minimizes how much collective traffic crosses BAG at all.

Training resilience across BAG

Training stays resilient end-to-end: when a replica fails, it is automatically excluded and the surviving replicas keep synchronizing across BAG. The direction of travel is longer distances still — training jobs spanning regions tens, hundreds, or thousands of km apart — and increasingly heterogeneous clusters mixing hardware generations and vendors.

Why it matters

  • A single training job that outgrows a region. BAG is the architectural move that turns "a cluster is a building" into "a cluster is a region (then multi-region)" — the networking precondition for gigawatt→multi-gigawatt training.
  • Lossless Ethernet at WAN distances. The in-place cable-length measurement is a concrete answer to the hardest problem of stretching RoCE across tens of km: headroom-by-measurement instead of headroom-by-worst-case.
  • Capacity-proportional routing with damping is a clean instance of reacting to degraded capacity, not just binary up/down — adjacent to static stability thinking.

Seen in (wiki)

  • sources/2026-10-19-meta-inside-prometheus-metas-journey-to-a-gigawatt-scale-supercluster — BAG as the super-spine that makes Prometheus a single machine, with hardware named. The Prometheus deep-dive details BAG as the multi-layer hub-and-spoke network connecting building-level fabrics across the cluster, spanning **2 km to

    100 km**, built on Arista 7800-series chassis running deep-buffered Broadcom Jericho3 silicon, delivering multiple Pbps per region-pair. It sits above the per-building fabrics (each interconnecting up to 20,736 GPUs via DSF→NSF), and the overall optics span 2×400G FR4 LITE short-reach pluggables inside buildings to 800G-ZR coherent optics beyond ~10 km. Running pretraining over this network required a dedicated optimization effort with Meta Superintelligence Labs "to ensure training was not slowed down, especially given speed-of-light constraints at these distances" — the distance-as-first-order-constraint theme this page describes, now attached to its primary-subject cluster (Prometheus).

  • sources/2026-10-12-meta-ocp-summit-2026-powering-ai-with-open-innovations-in-scale-up-scale-out-and-scale-across-networks — BAG as the scale-across super-spine; petabit scale, Prometheus/Hyperion, cable-length measurement, eBGP link-bandwidth signaling, collective scheduling, replica exclusion.
  • sources/2026-10-26-meta-networking-at-the-2026-ocp-global-summit-open-innovatio-ff1684df — OCP Summit 2026 talk roundup includes "Scaling AI Infrastructure Across DC and Beyond: Meta's Journey with BAG" (purpose, architecture, performance benefits + lessons from Prometheus deployment, explicitly covering platform fit, distance management, reliability, and capacity planning against a quickly-evolving AI ecosystem), a dedicated SRv6 backbone talk (why SRv6, data/control plane on FBOSS+SAI using shallow buffers at WAN distances, concept-to-deployment in months), and a cross-industry Scale-Across Networking Panel (Meta, Google Cloud, Microsoft Azure, Broadcom, Arista) comparing routing + optical-transport approaches across hyperscalers, silicon, and systems vendors.

Also referenced (not yet separately ingested): Meta's Feb-2026 "Building Prometheus: how backend aggregation enables gigawatt-scale AI clusters" post.

Last updated · 854 distilled / 2,448 read