Skip to content

META 2026-10-12

Read original ↗

OCP Summit 2026: Powering AI with Open Innovations in Scale-Up, Scale-Out, and Scale-Across Networks

Summary

At the 2026 Open Compute Project (OCP) Global Summit, Meta's Network Infrastructure team laid out a full-stack open-networking agenda organized around three distance domains of AI networking — scale-up (accelerator-to-accelerator within a rack/pod), scale-out (GPU-to-GPU across a training cluster), and scale-across (fabric-to-fabric across buildings, campuses, and regions). The headline hardware is IcePack, Meta's first liquid-cooled network rack, holding IceCube 102.4-Tbps switch blades (128×800G non-blocking, in Broadcom Tomahawk 6 and NVIDIA Spectrum-6 variants) — a 2× bandwidth step over the 51.2T generation. For scale-out, Meta introduced Flattened Planar Fabric (FPF), a multiplane backend fabric where each GPU attaches via eight independent 100G links to eight switches (one per plane), flattening the training fabric from a 3-tier to a 2-tier Clos and moving load-balancing from the switch into the NIC. For scale-across, the backend aggregation (BAG) petabit super-spine knits many spine fabrics into a single gigawatt-scale cluster (Prometheus ~1 GW in 2026, Hyperion toward 5 GW), with distance itself becoming a first-order constraint on lossless Ethernet buffer headroom. For scale-up, Tahan proves Ethernet can serve the in-rack accelerator interconnect, reusing the same lossless RoCE transport as the scale-out fabric. Two ecosystem initiatives round it out: OSPA (Open Switch Platform Architecture, co-led with Microsoft and Alibaba) to standardize the switch platform layer below SAI, and the maturing ESUN scale-up-Ethernet spec — plus open-source MetaRoCE and self-service FBOSS vendor onboarding.

Key takeaways

  1. The organizing frame is scale-up / scale-out / scale-across. Meta decomposes AI networking into three distance domains and argues Ethernet (with lossless RoCE) can now serve all three, unifying them under one NOS (FBOSS) and one transport. (Source: this article; see concepts/scale-up-scale-out-scale-across)

  2. IcePack / IceCube — liquid-cooled 102.4T switching. IcePack is Meta's first liquid-cooled network rack (on OCP's ORV3-HPRv1 standard); IceCube blades deliver 102.4 Tbps = 128×800G non-blocking, in two ASIC variants (Broadcom Tomahawk 6C and NVIDIA Spectrum-6), both running FBOSS over OCP SAI. A 2× bandwidth increase over 51.2T at the thermal density gigawatt clusters need. (Source: this article; see systems/icepack)

  3. FPF flattens the scale-out fabric and moves load-balancing to the NIC. Each GPU connects via 8 independent 100G links to 8 switches, one per network plane. This (a) builds larger non-blocking GPU clusters, (b) flattens 3-tier → 2-tier Clos (fewer hops, lower latency, less cost), (c) does NIC-level hardware load balancing — the NIC sprays every packet across all 8 planes instead of switch ECMP, and (d) runs a combined switch+host control plane so a host can steer traffic away from a downed plane. Built with NVIDIA, leveraging Spectrum-X Multiplane. (Source: this article; see systems/flattened-planar-fabric)

  4. BAG is a petabit scale-across super-spine, and distance is the new constraint. The backend aggregation (BAG) layer is a centralized Ethernet super-spine interconnecting many spine fabrics into one mega-cluster spanning buildings/campuses/regions; inter-BAG capacity between a region pair reaches tens of Pbps. It lets Prometheus (~1 GW, 2026) function as a single cluster and scales toward Hyperion (~5 GW). At tens of km, lossless Ethernet needs buffer headroom proportional to cable length, so FBOSS now measures actual cable length in place (timing probe traffic on an uncongested queue) and allocates headroom against the measurement rather than worst-case. Load balancing uses eBGP link-bandwidth signaling with damping; collectives get adaptive-routing + scheduling to avoid a latency cliff when traffic crosses BAG. (Source: this article; see systems/backend-aggregation-bag)

  5. SRv6 regional backbone on FBOSS over shallow-buffer silicon. Meta runs Segment Routing over IPv6 (SRv6) for its regional backbone, on open shallow-buffer switches powered by Cisco SiliconOne, all managed under FBOSS via OCP SAI — for flexible traffic engineering across the longest scale-across distances. (Source: this article)

  6. Tahan — Ethernet in the scale-up domain. Tahan is Meta's Ethernet-based scale-up switch for the MTIA 300 accelerator, built on the 51.2 Tbps Broadcom Tomahawk 5 (64×800G; half the ports used for scale-up backplane connectivity). Four Tahan switches interconnect 16 accelerators non-blocking. Crucially it reuses the same lossless RoCE transport (routed IPv6, PFC, AI-tuned QoS) as scale-out — unifying scale-up and scale-out under FBOSS. (Source: this article; see systems/tahan)

  7. Next-gen scale-up: 72 accelerators/rack on 102.4T, moving to ESUN. A preview 216×400G scale-up blade on Broadcom Tomahawk 6C interconnects 72 accelerators per rack (8 switches); a future generation adopts OCP ESUN as the scale-up data plane. (Source: this article; see systems/esun)

  8. ESUN matured to 1.0, heading to 1.1. The Ethernet for Scale-Up Networking initiative Meta founded in OCP in 2025 released the ESUN 1.0 Network Operator Requirements Base Specification, deliberately fabric-scoped (defines what the network must deliver, leaves GPU/accelerator implementation free) and coordinated with IEEE + UEC rather than duplicating them. ESUN 1.1 adds switch profiles and inter-packet-gap reduction. Partners: AMD, Arista, ARM, Broadcom, Cisco, HPE, Marvell, Microsoft, NVIDIA. (Source: this article; see systems/esun)

  9. OSPA standardizes the switch platform layer below SAI. SAI made the switch data plane portable across silicon, but the platform layer (power sequencing, thermal/fan, sensors, hardware inventory, out-of-band interfaces, firmware, diagnostics) stayed proprietary — an "onboarding tax" of months of duplicated bring-up per platform. OSPA (co-led by Meta, Microsoft, Alibaba) is a unified reference design for that layer; spec v1.0 is submitted to OCP. (Source: this article; see systems/ospa)

  10. Open ecosystem: FBOSS self-onboarding + open-source MetaRoCE. FBOSS was extended so switch vendors self-onboard their own platforms without Meta engineers in the loop (concurrent multi-platform development against a hyperscale-proven NOS). And MetaRoCE — Meta's RDMA transport for AI Ethernet — now ships open-source implementations. (Source: this article)

Architecture notes & numbers

Item Value
IceCube switch capacity 102.4 Tbps = 128×800G non-blocking
IceCube ASIC variants Broadcom Tomahawk 6C, NVIDIA Spectrum-6
Rack standard OCP ORV3-HPRv1 (IcePack, first liquid-cooled network rack)
Generational step 2× bandwidth over 51.2T platforms
FPF per-GPU uplinks 8 × 100G to 8 switches (1 per plane)
FPF topology change 3-tier NSF Clos → 2-tier Clos
BAG scale petabit; inter-BAG per region-pair tens of Pbps
Cluster scale Prometheus ~1 GW (2026); Hyperion toward 5 GW
Tahan ASIC Broadcom Tomahawk 5, 51.2 Tbps, 64×800G
Tahan topology 4 switches ↔ 16 accelerators non-blocking; 32×800G scale-up
Next-gen scale-up blade 216×400G on Tomahawk 6C; 72 accelerators/rack (8 switches)
Scale-up accelerator target 6.4 Tbps per accelerator (MTIA 300 system)
Regional backbone SRv6 on Cisco SiliconOne shallow-buffer, FBOSS/SAI

Caveats

  • Announcement/vision post at a summit, not an engineering deep-dive: FPF's control-plane mechanics, BAG's damping parameters, the cable-length probing algorithm, and ESUN wire-format details are described at a level, not specified.
  • Forward-looking items: the 72-accelerator ESUN-based scale-up blade is a preview; ESUN 1.1 and additional SiliconOne/silicon onboarding are in-progress; OSPA v1.0 is submitted to OCP (not yet a ratified standard).
  • Several numbers (Hyperion 5 GW, tens-of-Pbps inter-BAG) are capacity commitments / design targets, not measured production figures.

Source

Last updated · 854 distilled / 2,448 read