Skip to content

SYSTEM Cited by 1 source

Netflix Model Scoring Service (MSS)

Definition

Model Scoring Service (MSS) is Netflix's shared GPU inference backend — the component that handles model loading, batching, and GPU scheduling for models too large to run in-process on CPU. It sits behind the unified JVM-based serving system and supports XGBoost, TensorFlow, PyTorch, and LLMs behind a single interface, with NVIDIA Triton Inference Server as the compute layer managing GPU resources.

Architecture

MSS is structured as:

  1. Triton Inference Server (compute layer) — manages model loading, continuous batching, GPU scheduling, and multi-model concurrency.
  2. Java control plane (on top of Triton) — handles deployment orchestration, versioning, health checking, autoscaling, and multi-region rollout.
  3. vLLM (paved-path LLM engine) — integrated as the inference engine for LLM workloads within Triton, replacing TensorRT-LLM as of 2025.

Model authors package artifacts and configure deployment; the control plane provisions GPU instances, configures Triton, and orchestrates zero-downtime upgrades.

Access paths

Two paths reach MSS:

  • gRPC (existing) — via the JVM serving system, which handles routing, A/B, feature fetching, pre/post-processing, and logging.
  • Direct HTTP (newer) — OpenAI-compatible API for LLM-driven applications, exposed via Triton's OpenAI-compatible frontend (FastAPI + embedded Triton).

Deployment strategies

  • Red-Black — new version alongside old, phased traffic shift, atomic rollback on failure. Default for stable interfaces.
  • Versioned — independent per-(modelId, modelVersion) deployment for breaking interface changes. Temporary GPU cost overhead accepted for correctness.

Operational details

  • Model caching on FSx — models materialized on Amazon FSx at announcement time to avoid S3 download latency at boot.
  • Version pinning — Triton and vLLM versions pinned in the service image to prevent API-surface drift.
  • Unified metrics proxy — merges Triton and vLLM Prometheus metrics into a single /metrics endpoint (Triton's bridge only exposes 9 of 40+ vLLM metrics).

Seen in

Last updated · 585 distilled / 1,765 read