Skip to content

META 2026-10-28

Read original ↗

Meta Andromeda: Powering the Next Generation of Ads Retrieval Through Model-System Co-Design

Summary

Meta's 2026 update on Andromeda, the personalized ads retrieval engine (first stage of the retrieval → ranking funnel). Generative-AI creative tools grew the eligible-ad pool by "hundreds of millions of new ads", so the 2024 system (tens of millions of ads/sec per GPU) could no longer score the pool with a better model. Meta attacked the problem on three axes at once. (1) Model: a retrieval mixture-of-experts — an ensemble of specialized user towers as experts, a context-aware ad tower, and an attention-based MoE gating network that routes each ad to the top-K relevant user experts. It reaches near-ranking quality at ~300x fewer inference FLOPs than the 2024 model. (2) System: one fused GPU kernel for the whole per user-ad computation (embedding gather → MoE interaction → score), so intermediate tensors never touch HBM; plus co-location of targeting index matching, scoring and top-K selection on one GPU server, which cuts network traffic ~100x. (3) Hardware: a joint Meta–NVIDIA push toward the GH200 ceiling — 2x–5x per component, the gating kernel at ~85% of peak HBM bandwidth, INT8 for up to 2x more. Production result: >1 billion ads/sec per GPU server while scoring every eligible ad (a 100x lift over 2024) and +2.4% conversions on Facebook since 2025.

Key takeaways

  1. Bottleneck relocation at billion scale. At billions of ads/sec, the bottleneck is no longer arithmetic. Intermediate tensors between kernels cause "terabytes per second" of HBM traffic (concepts/memory-bandwidth-bound). Shipping candidates between CPU hosts and remote accelerators saturates the network link (concepts/network-bound-vs-compute-bound). Each fix exposes the next bound. (Source: this page)
  2. Retrieval MoE gives ranking-grade quality at retrieval cost. User side moves from one user tower to an ensemble of heterogeneous user-tower experts (sequential ad engagement, user-cohort representations, LLM-derived semantic features, cross-surface co-engagement). Ad side jointly encodes the ad with real-time serving context. The interaction "overarch" layer becomes an MoE attention-based gating network that activates only the top-K experts per ad. Net: ~300x fewer inference FLOPs than 2024 and >10B ads/sec on a single GH200 for model inference. This is the two-tower shape generalized to many user towers plus a learned router (concepts/mixture-of-experts).
  3. FLOP savings buy fusibility. The ~300x FLOP cut makes it possible to fuse both model ops and non-model system ops (embedding gathers, scoring) into one kernel; the 2024 launch fused only model ops. Intermediates stay in registers and thread-block shared memory and are never materialized to HBM (concepts/kernel-fusion).
  4. Collapse the system architecture to match the model. Before, index matching (advertiser targeting rules) and top-K selection ran on CPU hosts that talked to a remote GPU scoring service. Now the request carries only user features plus a targeting query to one GPU server, which runs index match → fused score → top-K locally and returns only the top-K. Network traffic drops ~100x; the system returns to a GPU-compute-bound regime. This is a hardware-level instance of patterns/co-located-inference-in-serving-layer.
  5. GPU targeting index is the hard part. Index matching evaluates complex advertiser targeting logic across "tens of feature categories" and must take real-time updates as ads are created, changed or expire. Meta redesigned the index structure for GPU-parallel execution "while preserving the full fidelity of production targeting semantics". After tuning it runs at >100B ads/sec — an order of magnitude above scoring — so it can be re-run across multiple retrieval pathways.
  6. Expressive model at dot-product speed. NVIDIA's optimized gating kernel went from 2.4B → >10B ads/sec (5.3x), at ~85% of peak HBM bandwidth, with latency equal to a plain dot-product scoring kernel on the same GPU. Meta's framing: "the additional model complexity is essentially free on Tensor Core hardware" — the roofline argument that a memory-bound kernel has idle compute to spend (systems/nvidia-tensor-core).
  7. INT8 doubles it again. INT8 concepts/quantization gave up to 2x over optimized FP16 kernels at accuracy on par with 16-bit.
  8. Data movement between components, not compute, was a large slice of end-to-end latency. Co-designing data layouts across index matching and scoring removed indirection; the full pipeline benchmarks at "multiple times" current production throughput on one server — headroom for deeper models and more retrieval pathways "without proportional infrastructure growth".
  9. Next: CPU:GPU ratio as a platform knob. On Grace Blackwell, Meta is studying how sensitive Andromeda-like workloads are to the CPU-to-GPU compute ratio.

Operational numbers

Metric Value
2024 Andromeda throughput tens of millions of ads/sec per GPU card
Required per-card throughput multi-billion ads/sec
2026 production pipeline >1B ads/sec per single GPU server, scoring every eligible ad (100x vs 2024)
Model inference / fused kernel >10B ads/sec on one GH200
Inference FLOPs vs 2024 model ~300x fewer
Network traffic after co-location ~100x less
Gating kernel speedup 2.4B → >10B ads/sec (5.3x), ~85% peak HBM BW
INT8 vs FP16 up to 2x throughput, accuracy on par
Index matching >2x speedup, >100B ads/sec
Per-component speedup (Meta–NVIDIA) 2x–5x
Business result +2.4% conversions on Facebook since 2025
2024 model capacity growth (prior post) four orders of magnitude

Architecture sketch

request {user features, targeting query}
        │  (compact payload — ~100x less network than shipping candidates)
        ▼
┌──────────────── single GPU server (GH200) ────────────────┐
│ index matching (GPU targeting index, real-time updates)    │  >100B ads/s
│        │ eligible ads                                       │
│        ▼                                                    │
│ fused scoring kernel: embedding gather → ad tower ctx →     │  >10B ads/s
│   MoE router → top-K user experts → relevance score         │  (registers /
│        │                     (no HBM intermediates)         │   smem only)
│        ▼                                                    │
│ top-K selection                                             │
└────────────────────────────┬───────────────────────────────┘
                             ▼
                   top-K candidates → ranking stage

Systems / concepts / patterns extracted

Caveats

  • Architecture-overview voice. No numbers on expert count, K, embedding sizes, candidate-pool size, latency budget, or fleet size.
  • "Single GPU" vs "single GPU server" is used loosely: the >10B figure is for the kernel on one GH200; the >1B production figure is per GPU server end to end.
  • The GPU targeting-index design (data structure, update path, consistency during real-time updates) is described only in one paragraph.
  • Index matching and GPU kernels were tuned with NVIDIA; the post does not say which of these optimizations are in production vs benchmark-only ("further gains expected as additional optimizations integrate into production").
  • Relationship to systems/silvertorch (also a Meta GPU-native, single-model retrieval stack with in-GPU filtering) and to KernelEvolve's >60% Andromeda kernel gains is not stated.
  • Partly promotional framing (NVIDIA partnership, advertiser value).

Source

Last updated · 854 distilled / 2,448 read