Skip to content

PATTERN Cited by 1 source

Batch-level constrained decoding

Intent

Scale custom constrained-decoding logic to high batch sizes by operating on the entire batch in one pass, eliminating the per-request GIL-serialized bottleneck.

Context

In vLLM V0, custom logits processors run per-request: the CPU processes each request's constraint logic sequentially because Python's GIL prevents parallelization. CPU time grows linearly with batch size, becoming the dominant latency contributor under production concurrency — invisible in single-request benchmarks.

Mechanism (vLLM V1)

  1. Implement the logits processor using vLLM V1's batch-level API — receives the full batch's logits and produces masks across all requests together.
  2. Rewrite the hot path in C++ with multi-threading to bypass the GIL.
  3. Model each constraint as a state machine that evolves with generated token history and emits token-eligibility masks at each step.
  4. Track batch membership changes explicitly via update_state(batch_update) — more complex than V0's per-request interface but necessary for dynamic batches.

Result

Logits processing time stays flat as batch size grows (vs. linear in V0).

Operational hardening required

  • Chunked prefill tracking: V1 may prefill across multiple steps; the processor must internally track whether prefill is complete.
  • Preemption recovery: Under memory pressure, vLLM evicts a request and reschedules with a different token list. The state machine must detect token-history shrinkage, reset, and reinitialize from the new prompt.

Seen in

Last updated · 585 distilled / 1,765 read