PATTERN Cited by 1 source

OpenAI-compatible frontend alongside gRPC

Intent

Expose both the ecosystem-standard OpenAI-compatible HTTP API and the existing internal gRPC interface from the same inference backend, enabling seamless migration from hosted LLM providers to self-hosted models.

Mechanism

1. Use NVIDIA's Triton OpenAI-compatible frontend (FastAPI-based) which starts an embedded Triton server.
2. The frontend wraps Triton in a TritonLLMEngine that converts OpenAI request schemas into Triton inference requests.
3. KServe HTTP/gRPC frontends remain enabled alongside, so the same Triton instance is accessible to the internal control plane over gRPC.
4. LLM models are NOT special snowflakes — every model (XGBoost ensemble or LLM) is reachable via the same gRPC path, reusing client libraries, health checking, and deployment pipelines.

Payoff

Graduating from a hosted model to a fine-tuned self-hosted one — for quality, latency, cost, or data privacy — becomes nearly seamless: same API, minimal code changes.

Patch required

Netflix found that the stock Triton OpenAI frontend silently drops response_format before reaching vLLM — callers requesting JSON output proceed without guided-decoding constraints and can receive malformed JSON with no error. They git-subtreed and patched the frontend to translate response_format into vLLM's guided-decoding parameters at request time.

Seen in

• sources/2026-07-17-netflix-in-house-llm-serving — Netflix's dual-API serving architecture