Skip to content

SYSTEM Cited by 1 source

Kunlun (GEM architecture)

Definition

Kunlun is the LLM-style model architecture introduced in the 2026 generation of GEM, Meta's central ads-recommendation foundation model. It replaces the previous Interformer architecture and is co-designed with the latest GPU hardware to make larger, deeper recsys models cost-efficient and to enable long-context learning over user engagement history. Meta reports Kunlun delivers 2× performance gains at the same computation scaling vs Interformer and an LLM-like scaling law at the largest pre-training scale with 2.5× model-complexity growth over earlier GEM versions (Source: sources/2026-10-13-meta-metas-generative-ads-model-gem-scaling). Reference: arXiv:2602.10016.

Why recsys needs its own architecture

Empirical scaling laws show transformer LLMs predictably improve with compute, parameters, and data — but recommendation models do not exhibit similar scaling with a plain transformer, because of massive heterogeneous features and shifting distributions. Kunlun is the architecture that "bends the scaling curve" for recsys: it standardizes an LLM-style framework while adapting attention, summarization, and cross-interaction to recsys data properties. It operates over two feature types:

  • Non-sequence features — user demographics, ad-content representations.
  • Sequence features — sequences of events, each composed of a set of features.

Three fundamental advancements

1. Standardized & unified architecture framework

Each layer consists of three parts:

  • Attention — customized attention + feed-forward networks capturing long dependency patterns in user behavior, adapting to data dynamics.
  • Summarization Component — simplified/enhanced architecture extracting critical user insights from sequences, bridging sequence and non-sequence structure.
  • Cross Interaction — expanded to a mixture-of-experts (MoE)-style design modeling complex interactions between sequence and non-sequence features at scale.

2. Scalable architecture through deep hardware co-design

  • Block Attention — replaces full self-attention with block-aligned sparse attention, reducing sequence-length complexity from quadratic → linear in window size; windows are multiples of the block size (kernel-friendly, no partial blocks).
  • Architecture Reformulation (pFFN → GDPA) — Per-Example Feed-Forward Networks (pFFN) adapt a traditional FFN to per-example distribution. Kunlun reformulates pFFN into attention operations via Generalized Dot-Product Attention (GDPA), unifying the model's many attention-like operations into a single GPU kernel for up to 2× forward speedup.
  • Scalable Cross Interaction via Expert Parallelism — the MoE-like cross-interaction uses model-expert parallelism: each GPU processes a subset of expert sub-networks in parallel.

3. Efficient sequence scaling

  • Component Skip Connection (CompSkip) — a computational skip mechanism that selectively activates sequence modules per layer and routes outputs through residual connections. By deactivating specific components (betting the network recovers quickly after re-activation) it reduces FLOPS by 43% with negligible quality loss.
  • Sequence Unification (OneSeq) — selectively unifies heterogeneous user sequences chronologically after event-feature aggregation, so a single unified sequence layer can process long sequences even when individual events carry different feature sets (e.g. a purchase-sequence event = ad ID + account ID vs a post-sequence event = actors
  • object ID — they can't be unified at the data-source level). Maximizes GPU efficiency via large tensor ops and minimizes operator fragmentation.

Scaling-law behavior

On the lightweight small-parameter GEM, Kunlun exhibits an LLM-like scaling law. Two reinforcing laws:

  • Prediction quality follows a power law with sequence length — predictable and consistent across sequence lengths.
  • Larger models benefit disproportionately from longer sequences — a virtuous cycle: scaling compute makes the model bigger (longer sequences more valuable), scaling sequence length makes it more accurate (justifies more compute).

GEM on Kunlun scaled the long engagement context 2× in production and demonstrated 25× long-context capability in infrastructure.

Domain expert framework

Kunlun is the substrate for GEM's cross-domain domain-expert framework: a shared expert learns a unified cross-domain representation while domain-specific experts capture surface-specific patterns, each benefiting from both. This is how GEM's single central model jointly predicts CTR across surfaces and CVR across objectives (see systems/meta-gem).

Caveats

  • Architecture given at the mechanism-sketch level (three advancements + named components); no layer topology, expert count, parameter split, or window-size numbers disclosed beyond the −43% CompSkip FLOPS and 2× pFFN→GDPA speedup.
  • "2× vs Interformer" and "2.5× model complexity" are Meta's own measurements; Interformer itself is named only as the predecessor, with no page on the wiki.
  • "25× long-context" is an infrastructure capability, distinct from the 2× scaled in production.

Seen in

Last updated · 854 distilled / 2,448 read