Skip to content

CONCEPT Cited by 1 source

Chunked prefill

Definition

Chunked prefill is an LLM inference scheduling technique where a request's prompt is prefilled over multiple engine steps (chunks) rather than processing the entire prompt in a single forward pass. This prevents a single long-prompt request from monopolizing the GPU and blocking other requests' decode steps.

Introduced in vLLM V1 as part of its scheduling redesign, chunked prefill allows the engine to interleave prefill chunks from new requests with decode steps from in-flight requests, improving latency fairness.

Statefulness challenge

For stateful custom logic that runs in the decode loop (e.g., constrained-decoding state machines), chunked prefill creates an ambiguity: a request can appear in multiple BatchUpdate notifications before its prefill is complete. Netflix found that vLLM's BatchUpdate API lacks granularity to distinguish partial from full prefill — requiring internal tracking within the logits processor to avoid advancing state prematurely.

Interaction with preemption

Under memory pressure, vLLM V1 may evict a partially completed request's KV cache and reschedule it later with a different prompt and output token list. Combined with chunked prefill, this means stateful processors must handle non-monotonic state transitions — the token history can shrink between decode steps.

Seen in

Last updated · 585 distilled / 1,765 read