PATTERN Cited by 1 source

Unified metrics proxy for merged observability

Intent

Merge multiple disjoint Prometheus metric sources from co-located inference components into a single /metrics endpoint, so existing dashboards and alerts work without per-component scraping changes.

Mechanism

1. A lightweight HTTP proxy runs alongside the inference stack.
2. It fetches Triton's server-level metrics via HTTP.
3. It reads vLLM metrics from disk using Prometheus's MultiProcessCollector (vLLM writes to PROMETHEUS_MULTIPROC_DIR as .db files).
4. Returns the combined output as a single Prometheus-compatible response.

Why it's needed

Triton and vLLM each expose metrics independently and neither is aware of the other. Triton's built-in bridge surfaces only 9 of 40+ vLLM metrics — missing critical operational signals:

• Token throughput
• KV cache utilization
• Prefix cache hit rates

Without the proxy, operators would need separate scrape targets and would miss the majority of inference-engine-level metrics.

Seen in

• sources/2026-07-17-netflix-in-house-llm-serving — Netflix's observability gap and proxy solution