CONCEPT Cited by 6 sources

Kernel fusion¶

Definition¶

Kernel fusion (also operator fusion) combines a sequence of GPU (or CPU SIMD) operations that would run as separate kernels into one kernel. Each separate kernel writes its output to device memory (HBM/DRAM) and the next kernel reads it back. A fused kernel keeps the intermediate values in registers and on-chip shared memory, so they are never materialized to HBM. It also removes per-kernel launch overhead.

Fusion matters most when a workload is concepts/memory-bandwidth-bound: the arithmetic is cheap but each intermediate tensor costs a round trip through memory. Fusion raises arithmetic intensity, which moves the kernel to the right on the roofline.

Why it matters at scale¶

Small per-item intermediates multiply by throughput. Meta Andromeda: at billions of ads/sec, "even a small intermediate tensor per ad multiplies into terabytes per second of high bandwidth memory (HBM) traffic". The memory traffic, not the math, was the dominant bottleneck in the multi-kernel pipeline (Source: sources/2026-10-28-meta-andromeda-powering-the-next-generation-of-ads-retrieval).

Scope of fusion¶

Model-op fusion — fuse ops inside the model graph (e.g. operators that share inputs). Often done by compilers such as torch.compile.
Model + system fusion — also fuse non-model work (embedding gather, filtering, scoring, top-K) into the same kernel. Andromeda 2024 fused only model ops; Andromeda 2026 fuses the whole per user-ad computation. A ~300x FLOP cut in the model made this feasible (Source: sources/2026-10-28-meta-andromeda-powering-the-next-generation-of-ads-retrieval).
Fusion with quantized data — dequantize-in-register inside the fused kernel (SilverTorch Int8 ANN; Dropbox low-bit inference).

Trade-offs¶

Fused kernels are custom code: harder to write, test and port across hardware. This cost is one motivation for agentic kernel generation (systems/kernelevolve).
Register / shared-memory pressure limits how much can be fused before occupancy drops.
Fusing system logic into the model kernel couples model and serving code.

Seen in¶

sources/2026-10-28-meta-andromeda-powering-the-next-generation-of-ads-retrieval — canonical instance: one kernel for embedding gather → MoE interaction → score;

10B ads/sec on one GH200, intermediates never hit HBM.
sources/2026-05-26-meta-silvertorch-index-as-model-a-new-retrieval-paradigm-for-recommendation-systems — fused Int8 ANN kernel, 2.2–14.7x faster than Faiss-GPU.
sources/2026-03-31-meta-adaptive-ranking-model-bending-the-inference-scaling-curve — "We fuse operators that share inputs" as an MFU lever.
sources/2026-03-03-pinterest-unifying-ads-engagement-modeling-across-pinterest-surfaces — fused embedding kernel improved inference latency.
sources/2026-02-12-dropbox-how-low-bit-inference-enables-efficient-ai — low-bit formats need custom fused kernels to pay off.
sources/2026-10-05-databricks-nearest-by-join-scaling-vector-search-in-databricks-runtime — fused blocked-GEMM operator turns a DRAM-bound pairwise distance into a compute-bound one.

concepts/memory-bandwidth-bound — the regime fusion fixes.
concepts/roofline-model — the analysis lens.
concepts/model-flops-utilization — fusion is a common MFU lever.
concepts/quantization — often combined with fusion.
systems/meta-andromeda-ads, systems/silvertorch, systems/kernelevolve.