Skip to content

PATTERN Cited by 1 source

State machine reset on preemption

Intent

Maintain correctness of stateful decode-loop logic when the inference engine evicts a request's KV cache and reschedules it with a potentially different token history.

Context

vLLM (and other inference engines with memory management) may preempt requests under GPU memory pressure — evicting a partially-completed request's KV cache and rescheduling it later. When this happens:

  • The rescheduled request may have a different prompt and output token list.
  • The stateful logic (e.g., constrained-decoding state machine) assumed the output token list grows monotonically.
  • The assumption is now violated — the token history can shrink between decode steps.

Mechanism

  1. At each decode step, compare the current output token list length against the previously tracked length.
  2. If the token history has shrunk (length decreased), detect this as a preemption event.
  3. Reset the state machine state entirely.
  4. Reinitialize from the new prompt + output token list to reconstruct the correct state.

Why this is non-obvious

Preemption is invisible to the logits processor API in normal operation. The design phase assumed monotonic progress. The failure mode only appears under production memory pressure with concurrent high-batch-size traffic.

Seen in

Last updated · 585 distilled / 1,765 read