SYSTEM Cited by 2 sources

NVIDIA Triton Inference Server¶

Definition¶

NVIDIA Triton Inference Server is an open-source ML serving runtime for production inference. Triton sits above compiled inference engines (TensorRT, TensorRT-LLM, ONNX Runtime, PyTorch, TensorFlow, Python custom backends) and exposes them as gRPC / HTTP endpoints with batching, scheduling, ensembles, model-versioning, and multi-model concurrency primitives.

Why it shows up on the wiki¶

Disclosed as the serving-runtime layer of Instacart's generative ads retrieval GPU stack:

"This new system leverages TensorRT-LLM for high-performance inference and is deployed on Nvidia's Triton Inference Server."

The full Instacart serving stack:

┌────────────────────────────────────────────────────┐
│  Go-native service shell (Griffin 2.0)              │
│  ├─ feature fetch / prompt assembly                 │
│  ├─ HTTP/gRPC client to Triton                      │
│  └─ retailer-partitioned index lookup post-decode   │
└─────────────────────────┬──────────────────────────┘
                          ▼
┌────────────────────────────────────────────────────┐
│  NVIDIA Triton Inference Server                     │
│  ├─ batching / scheduling / model versioning        │
│  └─ TensorRT-LLM backend (compiled decoder)         │
└─────────────────────────┬──────────────────────────┘
                          ▼
                  GPU (NVIDIA, SKU undisclosed)

The Go-native service shell handles request-handling, feature fetching, prompt assembly, and post-decode index lookup; Triton hosts the TensorRT-LLM compiled decoder; the GPU runs the actual beam-search autoregressive decoding.

Caveats¶

Specific Triton features used by Instacart (ensembles, business logic scripting, response cache) not disclosed.
Multi-tenancy / capacity-allocation / GPU-SKU details for Instacart's deployment not disclosed.

Netflix MSS usage (2026-07-17)¶

Netflix uses Triton as the compute layer inside their Model Scoring Service (MSS), managing model loading, batching, and GPU scheduling for XGBoost, TensorFlow, PyTorch, and LLMs behind a unified interface.

Key architectural details revealed:

Two packaging backends: The vLLM backend (JSON config → dynamic I/O spec generation, decoupled evolution) and the Python backend (explicit tensor specs, full execute() control for custom models).
OpenAI-compatible frontend: Triton's FastAPI-based OpenAI frontend starts an embedded Triton server, wraps it in a TritonLLMEngine, and serves alongside KServe gRPC frontends.
Version pinning required: Triton's vLLM backend is compiled against a specific vLLM API surface; drift causes load failures (e.g., Triton 25.09 importing vllm.engine.metrics, removed in vLLM 0.11.2).
Metrics gap: Triton's built-in Prometheus bridge surfaces only 9 of 40+ vLLM metrics; Netflix added a unified metrics proxy to merge both.
response_format silently dropped: The stock OpenAI frontend accepted but didn't pass response_format to vLLM — Netflix patched it to translate into guided-decoding parameters.

(Source: sources/2026-07-17-netflix-in-house-llm-serving)

Seen in¶

sources/2026-06-02-instacart-from-scoring-to-spelling-rebuilding-ads-retrieval-at-instacart — runtime layer for Instacart's generative ads retrieval GPU stack.
sources/2026-07-17-netflix-in-house-llm-serving — compute layer in Netflix's Model Scoring Service (MSS), paired with vLLM.

systems/tensorrt-llm — primary inference engine Triton hosts for LLM workloads.
systems/instacart-generative-ads-retrieval — production user.
systems/instacart-griffin-2 — Instacart's ML platform host.
patterns/gpu-serving-stack-tensorrt-llm-triton — the canonical pattern.