Skip to content

META 2026-10-19

Read original ↗

Inside Prometheus: Meta's Journey to a Gigawatt-Scale Supercluster

Summary

Meta's data-center-engineering deep-dive on Prometheus, its gigawatt-scale AI training supercluster in New Albany, Ohio — "hundreds of thousands of GPUs across multiple interconnected buildings … all tightly interconnected by miles of fiber and functioning as a single machine," with phase one already complete and used to train the Muse / Muse Wonder frontier models, scaling toward ~1 million GPUs. The article is the primary-subject companion to the networking-stack picture Meta drew at OCP Summit 2026: it narrates how a former conventional web-traffic site was rebuilt into a frontier-training cluster across three axes — multi-generation NVLink-72 hardware (Catalina GB200 → Clemente GB300 → upcoming San Miguel Vera Rubin), a two-tier building fabric + Backend Aggregation (BAG) super-spine stretching 2 km to >100 km, and a reliability program that drove a 99% reduction in interruptions (peak ~1,000 events per 8K GPUs/day in mid-2025 → a north-star of two interruptions per 8K GPUs/day ≈ ~100k GPU-hours MTBI). The load-bearing systems-design idea is that at this scale a single fault anywhere halts an entire synchronized training job, so goodput — not raw availability — is the master metric, and every layer (NVLink signal integrity, HBM reliability, fabric health, power transients, cooling) is engineered to shrink the 72-GPU failure domain the NVL72 architecture created.

Key takeaways

  • The 72-GPU scale-up domain is the new failure unit — a 9× bigger blast radius, answered with in-rack sparing. Moving from the H100-era NVL8 (8-GPU NVLink scale-up domain) to NVL72 "increas[es] the fault blast radius by 9X." Meta makes the 72-GPU rack the unit of capacity allocation: because training parallelism maps onto power-of-two group sizes, each NVL72 runs a 64-GPU slice and holds its remaining 8 GPUs as in-rack spares — a failed GPU is replaced immediately so the job restarts from its last checkpoint rather than waiting on repair, and the rack stays in the job. For a 32,768 active-GPU run this means provisioning 523 NVL72 racks = 37,656 GPUs (~13% above the active footprint), of which ~11% is the 64-of-72 slice geometry and only ~2% is held-in-reserve whole racks — a statically stable overprovision, not a reliability tax. (Source: this article)
  • Goodput, not availability, is the frontier-training metric. Muse Wonder achieved >85% goodput on pre-training and >90% on RL runs, at a wall-clock MTBI of ~47k GPU-hours, later raised past 54k GPU-hours via firmware fixes. Meta classes any fault — GPU silicon defect, fabric anomaly, storage bottleneck, thermal limit — as an interruption that forces a recovery cycle; the north star is two interruptions per 8K GPUs/day ≈ ~100k GPU-hours MTBI. (Source: this article)
  • Two-tier fault recovery matched to the parallelism map. Parallelism-Aware Fault Tolerance (PAFT) handles faults isolatable to one data-parallel replica: the faulty replica is taken offline, the rest repeat that step and continue, the compromised slice restarts on spare hosts and resyncs — "converting a potential total system halt into a minor, temporary reduction in aggregate throughput." When the blast area exceeds a replica, distributed checkpointing restarts the whole job from the last checkpoint; Meta uses delta checkpointing + in-memory staging to asynchronously save model + gradients + optimizer state to multi-tier storage with minimal throughput impact. (Source: this article)
  • A dedicated Back-End (BE) network + BAG super-spine turns many buildings into one machine. Latency/bandwidth-sensitive traffic concentrates within a data-parallel replica; higher-latency-tolerant traffic is scheduled between replicas — and the network is built to match: low-latency high-bandwidth building fabrics (each interconnecting up to 20,736 GPUs) aggregated by the Backend Aggregation (BAG) hub-and-spoke super-spine spanning 2 km to >100 km, built on Arista 7800 chassis with deep-buffered Broadcom Jericho3 silicon delivering multiple Pbps per region-pair. (Source: this article)
  • The building fabric evolved DSF → NSF as NICs gained endpoint reordering. The initial design used the Disaggregated Scheduled Fabric (DSF) — sub-packet multipath routing + packet reordering + virtual output queues + credit-based end-to-end flow control — to tame the few large, long-lived collective flows that collide under traditional load balancing. As Ethernet NICs gained endpoint packet reordering, Meta moved to a Non-Scheduled Fabric (NSF) on shallow-buffer 51.2T switches (Minipack3 with Dynamic Load Balancing; the NVIDIA Spectrum-4-ASIC Minipack3N with Adaptive Routing). Both run on FBOSS. (Source: this article)
  • Reliability was won by granular per-source decomposition under a co-engineering model with NVIDIA. Named categories and fixes: NVLink signal integrity (>600k NVLink connections; continuous telemetry + L11 diagnostic burns to isolate weak links before production); HBM/LPDDR memory reliability (fleet telemetry + refined HBM firmware, memory scrubbing, improved LPDDR ECC, pattern-based predictive retirement); NVLink switch & fabric health (millisecond-resolution monitoring + automated recovery without a job interruption); backend network events (hardware fault detection + planar topologies reroute around fiber/port failures). A colocated New Albany triage team, a March stability sprint, attribution-driven accountability (Meta owns scheduling/infra, NVIDIA owns firmware/signal/memory), and a mandatory post-repair quarantine underpin it. (Source: this article)
  • Hardware maturation at fleet scale is its own discipline. First-gen GB200 deployment surfaces defects hidden at low volume: firmware-bundle automation pushes fixes to >100k GPUs within hours (single-pod P90 upgrade cut from >7 h to 2 h); automated signal screening validates >600k NVLink paths before production traffic; power-delivery stress testing at the 1,400 W per-GPU load; precision liquid cooling quality gates on cold plates/manifolds; and pre-job health checks select healthy nodes for large jobs. (Source: this article)
  • Synchronized GPUs create multi-megawatt power transients; Dynamo Dimmer caps them softly. When tens of thousands of GPUs run compute/communication phases in unison they trigger 30-40% load swings within the cluster, and starting/pausing a job can shift tens of MW in seconds. Dynamo Dimmer, a cluster-wide power-management framework, gradually lowers GPU power limits as the cluster nears its electrical ceiling instead of hard-stopping work, reinforced by rack-level BBU modules, capacitor trays (CBU), facility-level BESS, and utility coordination. (Source: this article)
  • Heterogeneous cooling and a multi-generation roadmap. Prometheus mixes modular Air-Assisted Liquid Cooling (AALC) (free-air buildings + rapid-deployment structures) with Facility Liquid Cooling (FLC) in purpose-built AI buildings; Meta's newest DCs use closed-loop liquid cooling and open-source designs like IcePack via OCP. The roadmap mixes GB300 NVL72 with NVIDIA Vera Rubin NVL72 via the upcoming San Miguel platform, and informs the ground-up design of the next supercluster, Hyperion (Richland Parish, Louisiana). (Source: this article)

Hardware generations (operational numbers)

Platform Silicon HBM/GPU Peak compute NVLink BW Rack power Notes
Catalina (Ariel) NVIDIA GB200 NVL72 192 GB HBM3e (13.4 TB total) 720 PFLOPS FP8 / 1,440 PFLOPS FP4 130 TB/s ~120 kW (2-rack pair) 36 GPU/rack ×2 → 72-GPU NVLink domain; 2× ConnectX-7 400G → 800G scale-out/GPU; ORv3 HPR
Clemente NVIDIA GB300 NVL72 288 GB HBM3e (20.7 TB total; +50%) >1 exaFLOP FP4 130 TB/s (1.8 TB/s bidir/GPU) ~135 kW Full 72-GPU domain in a single rack; ConnectX-8 800G native; 2:1 GPU:CPU; AALC or FLC; 5-10% perf over GB200
San Miguel (upcoming) NVIDIA Vera Rubin NVL72 288 GB HBM4; 19.2 TB/s/GPU 2.52 EFLOPS FP4 / 1.26 EFLOPS FP8 per rack 216 TB/s (3 TB/s bidir/GPU) >170 kW, fully liquid-cooled 10-20% perf/watt over GB300

Fabric: each building fabric interconnects up to 20,736 GPUs; BAG spans **2 km →

100 km** on Arista 7800 / Broadcom Jericho3 at multiple Pbps/region-pair; optics range from 2×400G FR4 LITE short-reach pluggables inside buildings to 800G-ZR coherent for links beyond ~10 km. A single NVL72 domain carries 5,184 tuned differential-pair signaling links at 200 Gbps each.

Systems / concepts / patterns extracted

  • Systems — primary subject Prometheus supercluster (new page); extends BAG (primary-subject coverage of the super-spine it names), Catalina (the Ariel GB200 NVL72 variant + numbers), GB200 Grace Blackwell, DSF (→ NSF evolution), FBOSS. Clemente (GB300 NVL72), San Miguel (Vera Rubin NVL72), Dynamo Dimmer, Hyperion, NSF, Minipack3 / Minipack3N, Arista 7800, Broadcom Jericho3 named as plain text / systems: frontmatter — single-source, not described as their own architecture, below the Systems-gate bar.
  • Concepts — extends goodput (frontier-training master metric; MTBI arithmetic; interruption-rate decomposition), blast radius (the 72-GPU scale-up domain as the 9×-larger failure unit, contained by in-rack sparing), static stability (64-of-72 slice + reserve-rack overprovision absorbs failure without fetching new capacity), Clos / lossless Ethernet / scale-up/out/across (fabric), circular dependency (the bootstrapping tension gestured at in control-plane recovery). No new concept minted — every idea maps INTO existing vocabulary.
  • Patterns — staged rollout (the firmware/stability rollout + post-repair quarantine discipline); PAFT / delta-checkpointing / in-memory-staging / Dynamo-Dimmer-soft-power-capping / attribution-driven-co- engineering recorded as prose + tags (article-specific mechanisms, below the page-creation bar).

Caveats

  • First-party data-center-engineering voice with strong numbers but selective: goodput + MTBI + interruption-reduction + rack-power + the 523-rack provisioning example are concrete, but total Prometheus GPU count is given only as "hundreds of thousands … toward ~1 million", and there is no fleet-wide failure-rate breakdown by category, no per-category interruption attribution, no PAFT / checkpoint timing numbers, no Dynamo Dimmer control-loop detail, and no absolute power/cost figures.
  • PAFT and the Dynamo Dimmer / "Maximizing throughput" work reference external arXiv papers (not ingested here); mechanisms are at sketch level.
  • Several diagrams (BAG topology, GB200→GB200 interruption chart) are untranscribed images.

Source

Last updated · 854 distilled / 2,448 read