Skip to content

CONCEPT Cited by 7 sources

Tail latency at scale

Definition

"Tail latency at scale" names the failure mode where, as a system fans a single logical operation out across N hosts, the probability that at least one host is experiencing its worst-case latency during that operation approaches 1. Past some N, the tail of the host-latency distribution is the system's latency. Per-host p99s compose badly.

This is the effect Marc Brooker writes about in "The Tail at Scale" / GC-pause-dominated latency, and it is the forcing function behind many "why not JVM?" architectural decisions.

The math, informally

If each of N hosts has some probability p of being in a GC pause (or any tail event) at an arbitrary instant, and the operation must touch all N, then:

  • P(some host is paused during this op) ≈ 1 − (1−p)^N
  • For realistic p and growing N, this approaches 1 very quickly.
  • At this point median host behavior is irrelevant: the op inherits the worst host's latency distribution.

Seen in (case study): Aurora DSQL Crossbar simulation

The systems/aurora-dsql Crossbar fans reads across every journal (every host) to compose a global total order. The team simulated scaling the number of hosts while modeling occasional 1-second stalls (a realistic JVM GC-pause profile):

  • Target at 40 hosts: ~1,000,000 TPS, ~1s tail latency.
  • Observed at 40 hosts: ~6,000 TPS, ~10s tail latency.
  • Interpretation: "at scale, nearly every transaction would be affected by the worst-case latency of any single host in the system."

This result wasn't a tuning problem — it was architectural. The team concluded the only fix was to remove the variability source, which meant removing garbage collection from the hot path. That choice eliminated the JVM as a viable substrate for DSQL's data plane, even though JVM tuning was a path several DSQL engineers knew well.

Consequences for language choice

  • JVM tuning can reduce p but not eliminate GC-pause tail events; the (1−p)^N → 0 trap still applies at scale.
  • C / C++ removes GC but introduces memory-safety cost.
  • Rust removes GC and provides memory safety — which is why this concept shows up co-cited with concepts/memory-safety in the DSQL post.

It also motivates patterns/pilot-component-language-migration: because switching language is a one-way door, you prove the perf delta on the smallest isolated component (Adjudicator for DSQL) before committing the data plane.

Consequences for system architecture

Even before changing languages, the tail-at-scale result pushes designs toward:

  • Minimizing fanout per operation (if you can answer from 1 host, do).
  • Avoiding designs where every operation touches every host (DSQL's single-journal-per-commit forced this shape for reads — which is why removing GC variance mattered).
  • Hedging / request-cancellation strategies on the client side.

Seen in

Seen in (storage): EBS's entire history is a tail-latency story

systems/aws-ebs is a multi-tenant block-storage service whose customers' EC2 workloads inherit EBS's latency distribution. Spreading a hot tenant across many spindles reduces the spindle's outlier but widens the blast radius (see concepts/noisy-neighbor) — classic tail-at-scale compounding. Fifteen years of EBS design — patterns/full-stack-instrumentation, patterns/loopback-isolation, systems/nitro offload, systems/srd replacing TCP, custom systems/aws-nitro-ssd — is iterative variance removal at every layer. The latency arc: >10 ms avg IO (2008) → sub-ms consistent (io2 Block Express today), without ever taking the service offline.

See sources/2024-08-22-allthingsdistributed-continuous-reinvention-block-storage-at-aws.

Seen in (storage, object-level): S3 heat management

Warfield's FAST '23 keynote gives the object-storage flavor. A single HDD delivers ~120 IOPS (see concepts/hard-drive-physics); if a drive queues requests, the stall amplifies through any layer that was waiting on that I/O — metadata lookups, concepts/erasure-coding reconstructs where all k of k+m shards must return for a read to complete. Direct quote:

Hotspots at individual hard disks create tail latency, and ultimately, if you don't stay on top of them, they grow to eventually impact all request latency.

The S3 response is structural rather than per-query: concepts/heat-management via patterns/data-placement-spreading + patterns/redundancy-for-heat, underwritten by concepts/aggregate-demand-smoothing so any one tenant's burst is a small fraction of any one drive's load. The (1−p)^N trap is the same — at S3's disk count, some drive is always a little hot — but the flexibility to choose which drives to read from converts the problem from "avoid the hot drive" to "always have ≥ k non-hot drives among your read candidates."

See sources/2025-02-25-allthingsdistributed-building-and-operating-s3.

Seen in (ML / GPU training): grey failures in fanout

A distributed training step is synchronous fanout across ranks; the collective stalls on its slowest participant. One grey-failing GPU — thermally throttled, NIC packet loss — makes every rank wait. Checkpoint-and-full-restart is the customary response and wastes all peers' compute back to the last checkpoint. This is the exact same tail-at-scale math, with the supply side being grey-failure rather than GC pause. Structural answers combine concepts/grey-failure detection + patterns/partial-restart-fault-recovery so only the throttled rank is replaced — see HyperPod's training operator.

See sources/2025-08-06-allthingsdistributed-removing-friction-sagemaker-ai-development.

Seen in (scatter-gather search): OpenSearch coordinator fan-out

OpenSearch / Elasticsearch queries scatter-gather across N shards; the coordinator's end-to-end answer is max(per-shard latency) + collect tax. Figma's 2026 post "The Search for Speed in Figma" reports up to ~500 per-shard queries per user query in the initial configuration; cutting shards 450 → 180 decreased P50 in addition to p99 because the coordinator's scatter-gather cost had been dominating even the median — a direct application of tail-at-scale with fan-out width as the N. The coordinator-view latency was also hidden for months by a concepts/metric-granularity-mismatch: DataDog reported per- shard avg (8 ms) while the real coordinator avg was 150 ms. Pattern record: patterns/fewer-larger-shards-for-latency.

See sources/2026-04-21-figma-the-search-for-speed-in-figma.

Seen in (GPU inference serving): batching as a tail-stabilizer

Tail-latency-at-scale framing for GPU inference serving is not about per-host variance — it's about per-request queueing behaviour when traffic is spiky and the GPU is memory-bound. Sequential serving of short requests wastes GPU, which in turn caps throughput; the tail inflates under bursts because queued requests wait behind non-batched sequential inferences.

Voyage AI's 2025-12-18 result makes the point sharp: moving from no-batching + Hugging Face Inference to token-count-batched vLLM with padding removal delivered P90 end-to-end latency more stable during traffic spikes, even with fewer GPUs — and dropped by 60+ ms on some model servers under contention. Batching is the tail-stabilizer; autoscaling can't catch up to query-traffic bursts.

See sources/2025-12-18-mongodb-token-count-based-batching-faster-cheaper-embedding-inference.

Last updated · 178 distilled / 1,178 read