Skip to content

NETFLIX 2026-07-17

Read original ↗

In-House LLM Serving at Netflix

Summary

Netflix's AI Platform team describes their production LLM serving stack, built in-house atop vLLM and NVIDIA Triton Inference Server within the existing Model Scoring Service (MSS) infrastructure. Rather than consuming hosted LLM APIs, Netflix runs the full inference stack — model deployment, engine management, API surface, deployment strategy, and constrained-decoding logic — inside its production ML serving system. The post focuses on four key design decisions (engine selection, model packaging, API surface, deployment strategy) and a deep-dive on scaling constrained decoding from vLLM V0 to V1, revealing production gaps the design phase didn't anticipate.

Key Takeaways

  1. vLLM replaced TensorRT-LLM as the paved-path engine — by summer 2025, open-source engines closed the performance gap. vLLM won on operational fit: loads custom architectures without compilation, offers extensibility hooks for custom decoding, and is more debuggable. Practitioner familiarity cut research-to-production handoff cost.

  2. Two model-packaging modes create a decoupling spectrum — the vLLM backend (just JSON config + weights) decouples model from frontend evolution by generating I/O tensor specs dynamically; the Python backend gives full control for custom preprocessing/postprocessing but couples artifact to frontend API surface, requiring coordinated upgrades.

  3. Triton/vLLM version mismatch is a production hazard — Triton's vLLM backend is compiled against a specific vLLM API surface. Version drift (e.g. Triton 25.09 vs vLLM 0.11.2) causes backend load failures. The platform must pin compatible versions in the service image and prevent model authors from overriding.

  4. OpenAI-compatible HTTP API exposed alongside gRPC — adopting the ecosystem-standard OpenAI API as an additional frontend makes graduating from hosted to self-hosted models nearly seamless. Implementation reuses NVIDIA's Triton OpenAI-compatible frontend, git-subtreed and patched to translate response_format into vLLM guided-decoding parameters (was silently dropped by stock implementation).

  5. Two deployment strategies — Red-Black vs Versioned — Red-Black deploys alongside and shifts traffic phased (cheap, but breaks on I/O schema changes since consumer and model must migrate simultaneously). Versioned maintains independent per-(modelId, modelVersion) deployments allowing overlap at the cost of temporary GPU overprovisioning. Recommendation: embed variable configs (e.g. tensor shapes) into the model to stay on Red-Black; reserve Versioned for unavoidable breaking changes.

  6. Model caching on Amazon FSx eliminates cold-start pain — large LLMs are materialized on high-performance FSx at model-announcement time rather than downloaded from S3 at startup, bringing cold-start latency within scheduler tolerance.

  7. Constrained decoding scaled via batch-level V1 API — the initial per-request logits processor on vLLM V0 hit a GIL-induced serial bottleneck: CPU time grew linearly with batch size even though GPU work was efficiently batched. Migrating to vLLM V1's batch-level logits processing API + C++ multi-threaded hot path achieved flat processing time regardless of batch size.

  8. Preemption and chunked prefill break statefulness assumptions — vLLM V1's chunked prefilling can prefill a request over multiple engine steps; the BatchUpdate API lacks granularity to distinguish partial from full prefills (requires internal tracking). Under memory pressure, vLLM may evict a request's KV cache and reschedule with a different token list — breaking the state machine's monotonic-growth assumption. Detection + reset-from-new-prompt logic was added.

  9. Unified metrics proxy merges vLLM + Triton Prometheus endpoints — Triton's built-in bridge surfaces only 9 of 40+ vLLM metrics (missing token throughput, KV cache utilization, prefix cache hit rates). A lightweight HTTP proxy fetches Triton metrics via HTTP and vLLM metrics from disk (MultiProcessCollector), returning a merged /metrics endpoint.

  10. Architecture positions MSS as the shared GPU inference backend — the existing JVM-based serving system handles routing, A/B, feature fetching, logging; MSS handles model loading, batching, GPU scheduling. Two access paths: gRPC through the serving system (existing) and direct HTTP for newer LLM applications.

Operational Numbers

  • MSS supports XGBoost, TensorFlow, PyTorch, and LLMs behind a unified interface
  • vLLM V1 batch-level constrained decoding: processing time flat as batch size grows (Figure 3)
  • Triton built-in bridge exposes only 9 of 40+ vLLM metrics
  • Model caching: pre-staged on FSx at announcement time (vs S3 download at boot)

Caveats

  • The post acknowledges TensorRT-LLM may still have raw throughput advantages for standard models; vLLM was chosen for operational fit, not peak FLOPS.
  • The Python backend escape hatch will "likely remain necessary" for non-standard models — not all models can use the cleaner vLLM backend.
  • Versioned deployment's GPU cost overhead is explicitly called out as a trade-off accepted for correctness.

Source

Last updated · 585 distilled / 1,765 read