CONCEPT Cited by 1 source
Latency budget¶
Definition¶
A latency budget is the total wall-clock time allocated to a request path before the user experience degrades or a business SLO is violated. The budget is subdivided across layers (network, compute, data lookup, business logic) and each component must stay within its allocation.
In a checkout fraud-detection flow, the latency budget is the time between "tap pay" and "Approved" appearing — typically 50–200 ms. Every component in the path (model inference, feature lookup, network hop, business rules) consumes a slice of this budget.
Why it matters architecturally¶
- Drives technology choices. A 50 ms budget means you can't cold-start a database, can't tolerate 200 ms feature lookups, and must pre-warm connection pools.
- Makes network overhead visible. When model inference takes 10 ms but end-to-end takes 37 ms, the 17 ms gap (network, serialization, routing) is a first-class architectural concern — hence route optimization and data-plane shortcuts.
- Forces separation of latency-critical from latency-tolerant work. Profile updates (write path) are latency-tolerant; the next transaction read (hot path) is latency-critical. They share a table but have different consistency needs.
Example allocation (Databricks fraud detection)¶
| Component | p50 | Budget share |
|---|---|---|
| Feature lookup (Lakebase) | 8.9 ms | 33% |
| CatBoost inference | 0.4 ms | 1.5% |
| Container overhead | 0.2 ms | <1% |
| Network / data-plane hop | 17.4 ms | 64% |
| Business logic (profile + rules) | ~2 ms | ~1% |
| Total | ~27 ms | 100% |
The network hop dominates — which is why route optimization (data-plane shortcut) is the highest-leverage optimisation, not model tuning.
Seen in¶
- sources/2026-07-16-databricks-what-happens-in-the-milliseconds-after-you-tap-pay — 27 ms p50 / 37 ms p95 end-to-end for fraud scoring in a checkout flow. The latency budget breakdown reveals network overhead as the dominant consumer (64% of p50), justifying data-plane route optimization as the primary architectural investment.