Skip to content

META 2026-10-13

Read original ↗

Meta's Generative Ads Model (GEM): Scaling the Central Model and Compute Engine for Frontier Ads AI

Summary

The 2026 generation of GEM — Meta's recommendation-system foundation model behind ads across Facebook, Instagram, and Reels — is reframed as a single central foundation model + compute engine. Where the 2025 GEM and its 2026-08 training/architecture posts established GEM as the "central brain," this post describes its scale-up: a new LLM-style architecture (Kunlun, 2× the performance of the prior Interformer at equal compute), two post-training intelligence-transfer techniques (LoopFM structure representation + Solaris elastic/speculative precomputation), a shift from isolated per-domain optimization to unified cross-surface/cross-domain learning (one model jointly predicting CTR across all surfaces and CVR across all objectives), and a training stack that scaled total FLOPs 4× while doubling E2E efficiency to 20–25% MFU. Reported business impact: 18.9% conversion and 12% CTR improvement across ad surfaces, plus +5.8% / +4.1% / +1.6% conversion uplift on Facebook, Instagram, and off-site conversion for targeted users over the past year (Source: sources/2026-10-13-meta-metas-generative-ads-model-gem-scaling).

Key takeaways

  1. GEM is now a central model + compute engine, not per-domain. The 2026 generation shifts from isolated per-domain optimizations (separate FB / IG / Reels prediction systems) to one foundation model learning from signals across all of Meta's ads platforms — jointly predicting clicks and conversions across every surface and advertiser objective. 3.6 billion people use at least one Meta app daily; a per-surface, per-objective prediction system does not scale as surfaces and objectives multiply (Source: sources/2026-10-13-meta-metas-generative-ads-model-gem-scaling).

  2. Kunlun bends the recsys scaling curve. RecSys models historically don't scale like transformer LLMs (heterogeneous features, shifting distributions). Meta's new LLM-style architecture Kunlun — co-designed with the latest GPU hardware — delivered an LLM-like scaling law at the largest pre-training scale with 2.5× model complexity growth over earlier GEM, and 2× performance gains at equal compute vs the previous Interformer architecture (Source: sources/2026-10-13-meta-metas-generative-ads-model-gem-scaling).

  3. Two scaling laws reinforce each other. Prediction quality follows a power law with sequence length, and larger models benefit disproportionately from longer sequences — scaling compute makes the model bigger (makes long sequences more valuable), scaling sequence length makes the model more accurate (justifies more compute). GEM scaled long engagement-context length 2× in production and demonstrated 25× long-context capability in infrastructure (Source: sources/2026-10-13-meta-metas-generative-ads-model-gem-scaling).

  4. Post-training transfers 2× the intelligence of knowledge distillation alone. LoopFM compresses GEM's internal representations + sequential historical memory into a compact structure representation that downstream vertical models consume as long-context input — doubling intelligence transfer vs knowledge distillation at no additional foundation-model compute or latency (the representation is precomputed and shares computation with the soft-label generation already used for KD) (Source: sources/2026-10-13-meta-metas-generative-ads-model-gem-scaling).

  5. Solaris moves the expensive foundation-model inference in time, not out of the request. Evaluating GEM inside a live ad request is unaffordable, so Solaris speculatively precomputes user-ad embeddings for the top candidates a future request is likely to need and delivers them just-in-time on the serving hot path — "borrowing the idea of speculative decoding from LLM serving." The async jobs carry no latency budget (elastic background capacity), and coverage degrades hierarchically rather than dropping (fall back to the user's own recent embeddings across the same ad groups, or to behaviorally similar users) (Source: sources/2026-10-13-meta-metas-generative-ads-model-gem-scaling).

  6. Cross-domain unification is made tractable by cost-free data mixing + a domain expert framework. A virtual data-loading scheme mixes data from different pipelines without physically merging it (no extra storage), with mixture proportions controlled to approximate the real-time datastream order (mitigating domain-switch instability and recent-domain over-memorization). A domain expert framework — a shared expert learning a unified cross-domain representation plus domain-specific experts capturing surface-specific patterns, all built on the Kunlun MoE-style architecture — lets data-rich click tasks transfer into data-sparse conversion tasks (Source: sources/2026-10-13-meta-metas-generative-ads-model-gem-scaling).

  7. Training stack: 4× FLOPs, 2× efficiency, same co-design playbook. Over 12 months Meta doubled E2E training efficiency to 20–25% MFU while scaling total training FLOPs 4× — custom recommendation kernels (JFA, GDPA, Block-Attention) + mixed ultra-low-precision (MXFP8 attention + MLP) for compute efficiency, topology-aware 5D parallelism (2D FSDP + Expert Parallelism for dense params, Fully Sharded 2D Model Parallelism for sparse params) with SM-free collectives for scaling efficiency. "A co-design problem, not a software problem or a hardware problem alone" (companion blog: GEM Training at LLM Scale) (Source: sources/2026-10-13-meta-metas-generative-ads-model-gem-scaling).

Kunlun architecture (the new scaling core)

Kunlun replaces the previous Interformer architecture. It distinguishes two feature types — non-sequence features (user demographics, ad-content representations) and sequence features (sequences of events, each a set of features). Three fundamental advancements:

  1. Standardized & unified architecture framework — each layer has three parts:
  2. Attention — customized attention + feed-forward networks capturing long dependency patterns in user behavior, adapting to data dynamics.
  3. Summarization Component — a simplified/enhanced summarization architecture that extracts critical user insights from sequences, bridging sequence and non-sequence structure.

  4. Cross Interaction — expanded to a mixture-of-experts (MoE)-style design for large-scale modeling of interactions between sequence and non-sequence features.

  5. Scalable architecture through deep hardware co-design:

  6. Block Attention — replaces full self-attention with block-aligned sparse attention, dropping sequence-length complexity from quadratic → linear in window size; windows are multiples of block size (kernel-friendly, no partial blocks).
  7. Architecture Reformulation (pFFN → GDPA) — Per-Example Feed-Forward Networks (pFFN) adapt a traditional FFN to per-example distribution; Meta reformulated pFFN into attention operations via Generalized Dot-Product Attention (GDPA) to unify the model's many attention-like operations into a single GPU kernel, delivering up to 2× forward speedup.

  8. Scalable Cross Interaction via Expert Parallelism — the MoE-like cross-interaction uses model-expert parallelism, each GPU processing a subset of expert sub-networks in parallel.

  9. Efficient sequence scaling:

  10. Component Skip Connection (CompSkip) — a computational skip mechanism that selectively activates sequence modules per layer and routes outputs through residual connections, deactivating specific components (on the assumption the network recovers quickly after re-activation) to reduce FLOPS by 43% with negligible quality loss.
  11. Sequence Unification (OneSeq) — selectively unifies heterogeneous user sequences chronologically after event-feature aggregation, so a single unified sequence layer processes long sequences even when individual events have different feature sets (e.g. a purchase-sequence event = ad ID + account ID vs a post-sequence event = actors + object ID), maximizing GPU efficiency via large tensor ops and minimizing operator fragmentation.

Post-training: intelligence transfer at scale

The value of the foundation model depends on how effectively its intelligence transfers to the hundreds of vertical models serving ads in real time. Two mechanisms:

  • LoopFM — structure representation. KD is bandwidth-limited (downstream models get only a compressed view of GEM's intelligence per training step). LoopFM compresses GEM's intelligence into compact representations (summarized from important layers, describing each event GEM processes), structures them with deep historical memory (concatenating a user's prior GEM-generated representations into a sequential context window capturing cross-surface intent dynamics), and lets downstream models learn from this long-context representation at both inference and training time. The representation is precomputed and shares computation with the soft-label (KD) generation → no extra foundation-model cost. Doubles transfer effectiveness beyond KD.

  • Solaris — speculative / elastic precomputation. Solaris precomputes the user-ad representation from GEM just before vertical-model inference needs it. Three components: deliver fresh runtime knowledge (generate the user-ad embedding directly — real-time point-to-point knowledge of how a user will act on the exact ad candidate); speculate ahead of the request (as multi-stage ranking narrows a live request to its top candidates, evaluate those user-ad pairs with GEM in async background jobs with no latency budget + elastic capacity); enrich coverage hierarchically (when an exact user-ad embedding is unavailable, substitute an aggregate of the user's own recent embeddings across the same ad groups, or borrow from behaviorally similar users — degrade, don't drop).

Cross-surface / cross-domain optimization

GEM is now the first model to jointly predict CTR across all surfaces and CVR across all major optimization goals (off-site conversion, mobile app install), where previously each surface and objective had its own system. Learnings from engagement patterns across platforms transfer into conversion-behavior understanding. Two training-recipe innovations make heterogeneous large-scale learning tractable:

  • Cost-free data mixing — virtual data loading from different pipeline combinations without physically merging data (no extra storage); a data-mixture mechanism controls offline/online mixture proportions to approximate the real-time datastream order, mitigating domain-switch instability and recent-domain over-memorization.
  • Domain expert framework — a shared expert learns a unified cross-domain representation; domain-specific experts capture surface-specific patterns; each domain expert benefits from both. Built on the Kunlun architecture.

Operational numbers

  • 18.9% conversion improvement, 12% CTR improvement across Meta's ad surfaces and objectives.
  • +5.8% / +4.1% / +1.6% conversion uplift on Facebook / Instagram / off-site conversion for targeted users over the past year.
  • Kunlun: 2× performance gains at equal compute vs Interformer; 2.5× model complexity growth at the largest pre-training scale with an observed LLM-like scaling law.
  • Block Attention → sequence-length complexity quadratic → linear in window size.
  • pFFN→GDPA reformulation → up to 2× forward speedup.
  • CompSkip → −43% FLOPS at negligible quality loss.
  • Long engagement context scaled 2× in production, 25× capability demonstrated in infrastructure.
  • LoopFM → 2× intelligence-transfer effectiveness vs knowledge distillation, at no extra compute/latency.
  • Training: 4× total training FLOPs, E2E efficiency doubled to 20–25% MFU over 12 months.

Caveats

  • Vendor/first-party post with business-outcome framing; the headline uplift percentages are Meta's own A/B-attributed figures, not independently reproducible.
  • No fleet size, GPU vendor/model, parameter count, expert count, or wall-clock training time disclosed; "several thousand latest-generation GPUs" and "O(trillions) sparse / O(billions) dense" from the companion training post are the only magnitudes.
  • Kunlun / LoopFM / Solaris internals are given at the mechanism-sketch level with arXiv references (Kunlun arXiv:2602.10016, LoopFM arXiv:2605.29280, Solaris arXiv:2604.12110) but no reproducible layer topology, expert counts, or exact precompute scheduling.
  • "25× long-context capability" is an infrastructure demonstration, distinct from the 2× actually scaled in production.

Source

Last updated · 854 distilled / 2,448 read