Skip to content

DATABRICKS

Read original ↗

What happens in the milliseconds after you tap pay

Summary

A walkthrough of a real-time fraud-detection architecture built on Databricks, demonstrating how sub-50ms end-to-end latency is achieved for a payment approval flow. The system combines Model Serving with route optimization (a data-plane network-path shortcut), Lakebase (managed Postgres with autoscaling and scale-to-zero), and a FastAPI backend that runs business rules. Every layer instruments its own latency, producing an end-to-end latency waterfall from model container through network hop to application business logic.

Key takeaways

  1. End-to-end round trip is 27 ms at p50, 37 ms at p95 for the full fraud-scoring path: caller → route-optimized data plane → model container → Lakebase feature lookup → CatBoost inference → response. Well within a checkout latency budget.

  2. Feature lookup dominates model time, not inference. CatBoost prediction on a 12-feature vector takes 0.4 ms at p50; the Lakebase feature lookup takes 8.9 ms at p50. Implication: optimise the data path, not the compute path, for serving-dominated workloads.

  3. Route optimization as a data-plane shortcut. When enabled on a Model Serving endpoint, route optimization bypasses the standard control-plane request path and uses a shorter network hop via OAuth-authenticated data-plane queries. Network overhead drops to ~17 ms with only 4 ms p50→p95 spread — consistent and predictable.

  4. Connection pool + OAuth token rotation = double-check locking pattern. Because Lakebase authenticates via OAuth (client ID as Postgres username, token as password), pooled connections become invalid on token expiry. The architecture uses a _ensure_pool guard with double-check locking: fast path returns existing pool if token is unchanged; on rotation, a lock prevents concurrent pool rebuilds, and the old pool gets a 30-second grace period for in-flight queries.

  5. Two identical connection pools in separate processes. The model container maintains its own ThreadedConnectionPool to Lakebase (with background token refresh), independent from the FastAPI backend's pool. Same pattern, different lifecycle — a natural consequence of the model-serving isolation boundary.

  6. Autoscaling with scale-to-zero absorbs p95 spikes. Lakebase autoscales within a min/max range. The p95 jump from 8.9→13.9 ms on feature lookup correlates with connection-pool churn or brief scale-up events — still within latency budget, and exactly the spike autoscaling absorbs before it's user-visible.

  7. Business rules as a post-model layer that can override model approval. The model handles statistical risk (fraud probability); profile-based business rules (daily cap, international toggle) can override an "approved" model decision. Both read the same Lakebase table but serve different purposes — separation of concerns between ML and deterministic policy.

  8. Latency waterfall as first-class observability. Every layer reports timing: model_lookup_ms, model_inference_ms, model_total_ms, business_logic_ms, backend_total_ms. These compose into a per-request waterfall shown in the UI. The gap between round_trip_ms and model_total_ms isolates network overhead; the gap between backend_total_ms and model_call_ms isolates framework overhead.

  9. Benchmark methodology: 5,000 sequential requests, concurrency=1, 50ms delay. Purpose: isolate per-request latency anatomy (where milliseconds go at each layer) rather than load-test throughput. Appropriate for characterising the latency budget of a single transaction.

Operational numbers

Metric p50 p75 p90 p95
Feature lookup (Lakebase read inside model container) 8.9 ms 9.8 ms 11.7 ms 13.9 ms
CatBoost inference 0.4 ms 0.5 ms 1.6 ms 6.0 ms
Total model time (lookup + inference + container overhead) 9.5 ms 10.9 ms 14.9 ms 17.6 ms
End-to-end round trip (full data-plane call) 27.2 ms 29.6 ms 33.8 ms 37.3 ms
Network overhead (round trip − model time) 17.4 ms 18.5 ms 19.8 ms 21.1 ms

Architecture

Client → FastAPI Backend ──┬──→ Model Serving (route-optimized data plane)
                           │         └──→ Lakebase (feature lookup, 8.9ms p50)
                           │         └──→ CatBoost inference (0.4ms p50)
                           └──→ Lakebase (profile read + business rules)
  • Model Serving endpoint: fraud-detection-lakebase, CPU "Small" workload, single region
  • Data-plane query path: OAuth token exchange, SDK-managed, uses serving_endpoints_data_plane.query
  • Connection pool: psycopg2.pool.ThreadedConnectionPool, 3–10 connections
  • Auth: M2M OAuth — client ID as Postgres username, access token as password (no long-lived DB passwords)

Caveats

  • Benchmark uses sequential requests (concurrency=1), so contention effects, pool saturation under load, and scale-up latency are not tested.
  • Single region only — cross-region latency not characterised.
  • "Small" workload size — production deployments likely differ.
  • CatBoost model is deliberately small (12 features) — larger models would shift the bottleneck back to inference.

Source

Last updated · 585 distilled / 1,765 read