CONCEPT Cited by 1 source

Fused Int8 ANN search¶

Definition¶

Fused Int8 ANN search is the GPU-native approximate-nearest-neighbor search primitive used inside Meta's SilverTorch retrieval substrate (Source: sources/2026-05-26-meta-silvertorch-index-as-model-a-new-retrieval-paradigm-for-recommendation-systems). Two structural choices distinguish it from prior GPU-ANN libraries like Faiss-GPU:

Item embeddings stored in Int8 format inside the model graph — "cuts memory use roughly in half compared to typical 16 bits" — leveraging the GPU's dp4a (4×Int8 dot-product accumulate) instructions for compute, with "no measurable recall loss" relative to the full-precision baseline.
Search runs as a single fused GPU kernel — "reduces data movement and makes the retrieval stage cheap enough to return many more candidates."

It runs as a region of the SilverTorch retrieval network, not as a separate library or service.

Why "fused"¶

A general-purpose ANN library hands its outputs back to the caller — a query goes in, a top-K result list comes out, and the caller (typically a separate service) hands those results downstream. SilverTorch's ANN search lives inside the same model graph as the eligibility filter and scoring layer; "because the filter result is already inside the model, it can flow directly into ANN search without a separate service call," and search results flow into scoring inside the same forward pass. Multiple GPU kernels collapse into one, eliminating data-movement overhead between them — the literal sense of "fused" applied at retrieval-pipeline scope, not just within a single primitive.

Why Int8¶

The argument has two compounding parts (Source: sources/2026-05-26-meta-silvertorch-index-as-model-a-new-retrieval-paradigm-for-recommendation-systems):

Memory. Int8 ≈ ½ the storage of typical 16-bit (FP16 / BF16) embeddings. For a fixed HBM budget, twice the candidate pool fits on-chip — and HBM residency is the constraint that determines how widely the retrieval funnel can open.
Compute. GPUs accelerate Int8 dot products via the dp4a instruction (4 × Int8 multiplies + accumulate per cycle). This is the substrate-level performance win that compounds with the kernel-fusion structural win.

Recall-vs-precision tradeoff disclosed: "Our Int8 quantized ANN search shows limited quality loss compared to brute force while significantly improving serving performance ... in practice, we observe no retrieval recall loss with 64 probes and top-2048." This is a specific operating point — the post does not bound recall at smaller probe counts or larger top-K.

Performance vs Faiss-GPU¶

Metric	Faiss-GPU	Fused Int8 ANN (SilverTorch)
Compute-cost efficiency vs CPU baseline	5.9×	20.9× (overall stack incl. filter + scoring)
Per-kernel speedup (this primitive only)	baseline	2.2–14.7×
Maximum top-k	2,048	hundreds of thousands
Neural reranking on retrieval candidates	✗	✓
Multi-task scoring	✗	✓

The 2.2–14.7× per-primitive speedup is the contribution of fused-Int8-ANN alone; the rest of the SilverTorch advantage decomposes into the Bloom index filter (291–523× over CPU inverted index) and the probe-then-filter co-design (30× filter compute reduction).

Why a redesign, not a port¶

Faiss-GPU is a high-quality library — but it's "built to find nearby items" in isolation, oriented around exposing a clean Python / C++ ANN-only API. "Recommendation systems need more than a small nearest-neighbor lookup. They often need to pull back a much larger pool of candidates so later stages can make better relevance decisions." The redesign's value is in (a) the larger top-K capability, (b) the in-graph fusion with adjacent stages, and (c) the freedom to compose with in-graph eligibility filtering under the patterns/gpu-native-retrieval-primitive-redesign pattern.

The post is explicit about the philosophy: "once retrieval components live inside one PyTorch model, co-design becomes possible, and that co-design is what unlocks the gains."

Relationship to existing wiki ANN material¶

concepts/ann-index catalogues the ANN-index family (HNSW / IVF / IVFPQ / DiskANN / SPANN / SPFresh) and the production-engineering observation that ANN indices ship on rebuild cadences slower than model rollouts. Fused Int8 ANN inverts that pattern — the index is part of the model, so freshness is a streaming-weight-update problem rather than a rebuild-cadence problem.
systems/faiss is Meta's open-source ANN library. SilverTorch supersedes Faiss-GPU inside the recsys retrieval surfaces it now powers, not Faiss-the-library across all Meta search/retrieval workloads (e.g., systems/meta-groups-scoped-search continues to use Faiss).
concepts/selective-fp8-quantization (Meta MARM 2026-03-31) is the sibling quantization-as-architectural-decision instance — selective-FP8 in MARM, end-to-end-Int8 in SilverTorch, both "only on benchmark-verified-precision-tolerant layers / paths" and both leveraging GPU instruction-set acceleration.

Caveats¶

"No measurable recall loss" at 64 probes / top-2048 is the disclosed operating point; smaller-probe or larger-top-K recall behavior is not bounded.
The exact Int8 quantization scheme (per-row vs per-column vs per-tile scales, calibration distribution, asymmetric vs symmetric) is not disclosed in the post — likely in the SIGIR 2026 paper (arXiv:2511.14881).
The 2.2–14.7× range over Faiss-GPU is a wide band; the post does not enumerate which configurations sit at the low vs high end.

Seen in¶

sources/2026-05-26-meta-silvertorch-index-as-model-a-new-retrieval-paradigm-for-recommendation-systems