CONCEPT Cited by 1 source

GIL bottleneck in inference pipelines

Definition

The GIL (Global Interpreter Lock) bottleneck in inference pipelines occurs when custom Python logic that runs per-request inside the model's decode loop cannot be parallelized across requests in a batch, causing CPU time to grow linearly with batch size even though the GPU forward pass is efficiently batched.

Netflix's experience

Netflix's constrained-decoding logits processor on vLLM V0 demonstrated this pattern:

1. GPU produces logits for the whole batch.
2. CPU copies logits from GPU and waits for the transfer.
3. Constraint logic runs sequentially for each request — the GIL prevents Python from parallelizing per-request work.
4. End-to-end latency becomes CPU-bound under realistic concurrency.

This bottleneck is invisible in single-request benchmarks and only surfaces under production batch sizes.

Resolution

The structural fix required two changes:

1. Batch-level API (vLLM V1) — logits processing operates on the entire batch at once rather than iterating per-request.
2. C++ hot path with multi-threading — the compute-intensive constraint evaluation is reimplemented in C++, stepping around the GIL entirely.

The result: logits processing time stays flat as batch size grows.

Seen in

• sources/2026-07-17-netflix-in-house-llm-serving — primary description of the GIL scaling bottleneck in constrained decoding