Skip to content

CONCEPT Cited by 1 source

Hot-node data amplification

Hot-node data amplification is the failure mode where consistent hashing + power-law traffic distribution causes a small number of nodes to receive orders of magnitude more data than their peers, compounded by data multiplication during redistribution — multiple upstream instances all routing aggregators for the same hot key to a single owner.

The mechanism

  1. Power-law traffic: popular services (authentication, recommendations) are called by hundreds of upstream services, generating 100× more flow records than typical services.
  2. Consistent hashing routes by destination: all flow records for a given destination route to the same "owner" instance.
  3. Data amplification during redistribution: if the hot service has flow records spread across 10 instances (because many clients call it), all 10 instances route their aggregators for that destination to one owner — the volume multiplies at the collection point.
  4. GC vicious cycle: the hot instance allocates objects faster than it can process → memory pressure → frequent GC pauses → GC consumes CPU → processing slows → memory pressure increases → instance goes DOWN → cascading as load redistributes.

Netflix observed instances handling 100× the flow records of their peers, with GC consuming more CPU than business logic. (Source: sources/2026-07-13-netflix-building-service-topology-at-scale-architecture-challenges)

Why it's worse than simple hot keys

Simple hot-key problems (one key gets more traffic) affect one shard. Data amplification is worse because:

  • The hot instance receives aggregators from many upstream instances, not just raw data from one source.
  • The amplification ratio scales with cluster size (more upstream instances = more aggregators converging on the owner).
  • The failure cascades — when the hot instance dies, its load redistributes and creates new hot instances.

Solutions

  • patterns/graduated-redistribution-for-skewed-data — multi-stage pipelines where each stage re-hashes on different keys, spreading load across multiple distribution points.
  • Pre-aggregation at upstream stages (compress before routing).
  • Splitting enrichment (I/O-heavy) from resolution (CPU-heavy) into separate stages.

Seen in

Last updated · 585 distilled / 1,765 read