PATTERN Cited by 1 source

Latency waterfall observability

The pattern

Each layer in a request path instruments and reports its own internal timing as structured fields in the response payload. Upstream layers aggregate these fields into a per-request latency waterfall — a breakdown showing exactly where milliseconds were spent, with gaps between layers exposing network/serialisation overhead.

Mechanism

Response {
  model_lookup_ms:    8.9    // measured inside model container
  model_inference_ms: 0.4    // measured inside model container
  model_total_ms:     9.5    // sum + container overhead
  business_logic_ms:  2.1    // measured in backend
  model_call_ms:     27.2    // wall-clock at backend for entire model call
  backend_total_ms:  29.3    // backend start-to-finish
}

Derived insights from gaps: - model_call_ms − model_total_ms = network overhead (routing, serialization, data-plane hop) - backend_total_ms − model_call_ms = framework overhead (FastAPI routing, serialization)

Why this works better than traces alone

1. No trace infrastructure required. Timing is in the response payload — any client can see it.
2. Per-request granularity. Every single transaction carries its own breakdown, not just sampled traces.
3. User-facing. The waterfall renders in the UI so developers (and end users during demos) see exactly where time goes.
4. Isolation diagnosis. When route optimization is toggled, the network-overhead gap changes — quantifying the data-plane shortcut without touching model code.

When to use

• Real-time ML serving where latency budget is tight (sub-100ms) and you need to know which layer to optimize.
• Multi-hop request paths (client → API → model → database) where aggregate latency hides the bottleneck.
• A/B testing infrastructure changes (e.g., route optimization on/off) — the waterfall quantifies the effect per-request.

Trade-offs

Advantage	Cost
Zero-dependency observability (no OTel collector needed)	Response payload grows with each instrumented layer
Every request is its own trace	Not a substitute for distributed tracing across microservices
Natural fit for demo/debug UIs	Must trust each layer to measure honestly (clock skew in multi-process)

Seen in

• sources/2026-07-16-databricks-what-happens-in-the-milliseconds-after-you-tap-pay — The Databricks retail-app fraud-detection system returns model_lookup_ms, model_inference_ms, model_total_ms, business_logic_ms, and backend_total_ms in every transaction response. The UI renders these as a latency waterfall, and the network-overhead gap quantifies route optimization's effect.