CONCEPT Cited by 1 source
Prefill-only inference¶
Definition¶
Prefill-only inference is an LLM serving mode where the model processes the input prompt in a single forward pass (the prefill phase) and extracts a representation — without performing any autoregressive token-by-token decoding. The model never enters the decode phase.
This is specifically useful for scoring or embedding workloads where the output is a numeric vector (not generated text). The cost profile shifts from O(prompt + generated_tokens) to O(prompt) only, eliminating the sequential decode bottleneck.
Why It Matters¶
Autoregressive decoding over large candidate sets is prohibitively expensive for recommendation workloads. If a model must decode N candidate IDs one-by-one, the cost is linear in N. Prefill-only mode:
- Processes the prompt once — a single parallel forward pass across all tokens.
- Extracts a pooled representation from the last-layer hidden states.
- Scores the entire candidate set via a scoring head (dot product or MLP against learned item embeddings) in a single matrix operation.
The distinction from standard embedding inference is that prefill-only inference here still leverages a generative model's architecture and training signal — the model was trained with next-token-prediction objectives — but at inference time the decoding path is never invoked.
Trade-offs¶
| Property | Prefill-only | Autoregressive |
|---|---|---|
| Latency | Single forward pass + matmul | Sequential per-token decoding |
| Cost | O(prompt_length) | O(prompt_length + output_length) |
| Output | Numeric scores / embeddings | Generated tokens |
| Flexibility | Fixed scoring function | Arbitrary text output |
| KV cache | Populated once, discarded | Grows with output length |
Seen in¶
- sources/2026-07-30-netflix-genrec-towards-llm-native-recommendation — Netflix GenRec runs in prefill-only mode on vLLM: "the model consumes the prompt once and scores the entire candidate set in a single forward pass, with no token-by-token decoding".