Skip to content

SYSTEM Cited by 1 source

Prometheus (Meta gigawatt-scale AI supercluster)

Disambiguation. This is Meta's AI training supercluster in New Albany, Ohio. It is unrelated to the CNCF monitoring system Prometheus (time-series database).

Prometheus is Meta's gigawatt-scale AI training supercluster — "hundreds of thousands of GPUs across multiple interconnected buildings … all tightly interconnected by miles of fiber and functioning as a single machine." Phase one is complete and was used to train the Muse / Muse Wonder frontier models; Meta is scaling it toward ~1 million distributed GPUs, all operating in lockstep. It is built across a mix of traditional data-center buildings, AI-optimized buildings, and rapid-deployment structures on a former conventional web-traffic site.

Prometheus is the physical cluster that sits beneath Meta's OCP-Summit-2026 networking stack (see OCP Summit 2026): the BAG super-spine, FPF, and the scale-up/out/across framing all describe how Prometheus is wired.

Multi-generation NVL72 hardware

Every platform uses the NVL72 rack architecture — 72 GPUs act as a single accelerator (one NVLink scale-up domain) within the fabric:

  • Catalina (Ariel GB200 NVL72) — first platform; see Catalina + GB200. Two-rack config, 36 GPUs/rack over NVLink → 72-GPU domain; 192 GB HBM3e/GPU (13.4 TB), 720 PFLOPS FP8 / 1,440 PFLOPS FP4, 130 TB/s NVLink, ~120 kW per two-rack pair; 2× ConnectX-7 400G NICs → 800G scale-out per GPU (deployed early to avoid the GB200's 400G-per-GPU scale-out imbalance).
  • Clemente (GB300 NVL72) — second platform. 288 GB HBM3e/GPU (20.7 TB, +50%), >1 exaFLOP FP4, 1.8 TB/s bidirectional NVLink/GPU, ~135 kW/rack; packs the full 72-GPU domain into a single rack via a 2:1 GPU:CPU ratio and ConnectX-8 SuperNICs delivering 800G natively; AALC or FLC cooled; 5-10% faster than GB200. (Single-source sub-platform — recorded here, no separate page.)
  • San Miguel (Vera Rubin NVL72, upcoming) — 288 GB HBM4/GPU @ 19.2 TB/s, 2.52 EFLOPS FP4 / 1.26 EFLOPS FP8 per rack, 216 TB/s NVLink (3 TB/s bidir/GPU), fully liquid-cooled >170 kW/rack, 10-20% better perf/watt than GB300. Integrates into Prometheus's existing fabric. (Single-source sub-platform — recorded here, no separate page.)

The 72-GPU failure domain

The NVL8 (H100) → NVL72 transition expands the scale-up domain from 8 to 72 GPUs, increasing the fault blast radius by 9×. Meta's answer makes the 72-GPU rack the unit of capacity allocation:

  • Training parallelism maps onto power-of-two group sizes, so each NVL72 runs a 64-GPU slice and keeps its remaining 8 GPUs as in-rack spares. A failed GPU is replaced immediately so the job restarts from its last checkpoint rather than waiting on hardware repair, and the rack stays in the job.
  • Whole-rack faults are covered by holding ~2% additional racks in reserve.
  • A 32,768 active-GPU run → 523 NVL72 racks = 37,656 GPUs (~13% over the active footprint), of which ~11% is the 64-of-72 slice geometry and only ~2% is a reliability reserve — a statically stable overprovision that absorbs failure without fetching new capacity.
  • Physical scale of one domain: 5,184 tuned differential-pair signaling links at 200 Gbps, each crossing GPU SerDes, compute-tray PCBs, cable cartridges, and switch-tray interfaces.

Networking — building fabric + BAG super-spine

A dedicated Back-End (BE) Ethernet network carries direct GPU-to-GPU traffic, designed to match the training hierarchy (latency/bandwidth-sensitive traffic within a data-parallel replica; higher-latency-tolerant traffic between replicas):

  • Building fabrics interconnect up to 20,736 GPUs each. The original design used the Disaggregated Scheduled Fabric (DSF) (sub-packet multipath + reordering + virtual output queues + credit-based end-to-end flow control) to tame collective flows that collide under traditional load balancing; as NICs gained endpoint packet reordering, Meta moved to a Non-Scheduled Fabric (NSF) on shallow-buffer 51.2T switches (Minipack3 with Dynamic Load Balancing; Spectrum-4-ASIC Minipack3N with Adaptive Routing). Both run on FBOSS.
  • The Backend Aggregation (BAG) hub-and-spoke super-spine knits building fabrics into one cluster across 2 km to >100 km, on Arista 7800 chassis with deep-buffered Broadcom Jericho3 silicon at multiple Pbps/region-pair. Optics span 2×400G FR4 LITE short-reach inside buildings to 800G-ZR coherent beyond ~10 km.

Reliability — goodput as the master metric

At this scale "a single fault anywhere can halt an entire training job," so goodput governs:

  • Muse Wonder: >85% pre-training goodput, >90% RL goodput, wall-clock MTBI ~47k GPU-hours, raised past 54k GPU-hours by firmware fixes.
  • Mid-2025 peak ~1,000 interruptions per 8K GPUs/day → 99% reduction over a year. North star: two interruptions per 8K GPUs/day ≈ ~100k GPU-hours MTBI.
  • Two-tier fault recovery: PAFT (Parallelism-Aware Fault Tolerance) isolates a fault to one data-parallel replica (take it offline, the rest repeat the step, the slice restarts on spares and resyncs — a temporary throughput dip, not a halt); for larger blast areas, distributed checkpointing restarts the job from the last checkpoint, with delta checkpointing + in-memory staging keeping saves cheap.
  • Per-source decomposition under a co-engineering model with NVIDIA (attribution-driven: Meta owns scheduling/infra, NVIDIA owns firmware/signal/ memory): NVLink signal integrity (>600k connections; L11 diagnostic burns), HBM/ LPDDR reliability (firmware tuning, memory scrubbing, predictive retirement), NVLink switch/fabric health (millisecond monitoring + automated recovery), backend network events (planar reroute). A colocated New Albany triage team, a March stability sprint, and a mandatory post-repair quarantine complete the program.

Hardware maturation + power + cooling

  • Fleet maturation: firmware-bundle automation pushes fixes to >100k GPUs within hours (single-pod P90 upgrade >7 h → 2 h); >600k NVLink paths signal-screened before production; power-delivery stress testing at 1,400 W/GPU; precision-liquid-cooling quality gates; pre-job health checks.
  • Power transients: synchronized GPUs cause 30-40% load swings; job start/stop shifts tens of MW in seconds. Dynamo Dimmer gradually caps GPU power as the cluster nears its electrical ceiling (soft-cap, not hard-stop), reinforced by rack BBU, capacitor trays (CBU), facility BESS, and utility coordination. (Single-source framework — recorded here, no separate page.)
  • Cooling: a mix of modular AALC (free-air + rapid-deployment structures) and FLC (purpose-built AI buildings); newest DCs use closed-loop liquid cooling; designs like IcePack shared via OCP.

Beyond Prometheus

The roadmap mixes GB300 NVL72 with NVIDIA Vera Rubin NVL72 (via San Miguel), and the blueprint informs the ground-up design of Hyperion, Meta's next supercluster in Richland Parish, Louisiana (closed-loop cooling, full grid connectivity, next-gen network designs). (Hyperion named only — no page.)

Seen in (wiki)

Last updated · 854 distilled / 2,448 read