PATTERN Cited by 1 source

Prefill-only scoring for LLM recommenders

Pattern

Use an LLM backbone in prefill-only mode — process the verbalized user context in a single forward pass to extract a pooled representation, then score the full catalog via a scoring head (dot product with learned item embeddings). Never invoke autoregressive decoding.

Problem

Autoregressive decoding over large candidate sets is cost-prohibitive. If the model must generate token-by-token for each of N candidates, serving cost scales linearly with catalog size and is dominated by the sequential decode latency.

Solution

1. Prefill the prompt (verbalized user history + context) in one parallel forward pass.
2. Pool the final hidden states into a single representation h.
3. Score all catalog items via h × E^T (where E is the item embedding matrix) in a single matmul.
4. Sort scores to produce the recommendation ranking.

The decode engine (KV cache growth, speculative sampling, beam search) is never activated.

Consequences

• Serving cost proportional to prompt length only, not catalog size.
• Compatible with existing LLM serving infrastructure (vLLM, Triton) — just run the prefill step.
• Context compaction directly reduces cost (every token saved = proportional cost saving).
• Loses the ability to generate explanations or conversational outputs (trade-off: add a separate decode path if needed).

Seen in

• sources/2026-07-30-netflix-genrec-towards-llm-native-recommendation — GenRec on Netflix's vLLM stack: "we run in prefill-only mode: the model consumes the prompt once and scores the entire candidate set in a single forward pass".