CONCEPT Cited by 1 source
Hotspot¶
A hotspot is a performance-degraded node in a distributed system that takes a disproportionate share of data storage or request traffic. Source's definition (sources/2023-02-22-highscalability-consistent-hashing-algorithm):
"Hotspot: a performance-degraded node in a distributed system due to a large share of data storage and a high volume of retrieval or storage requests."
Two root causes¶
- Uneven key distribution — the partitioning scheme gave one node more keys than others (raw consistent hashing without virtual nodes produces this via arc-length variance).
- Skewed key popularity — one key is much hotter than the others, so even a perfectly-balanced partitioning concentrates traffic on whichever node owns it (the "single popular data object" case).
Virtual nodes fix (1) but not (2); bounded-load consistent hashing was specifically built to address (2).
Why hotspots are dangerous¶
A hotspot node's CPU / memory / network saturates; P99 latency climbs; health checks start to fail. When the hotspot node is marked unhealthy, its traffic is shed to the remaining nodes — which are already at or near capacity — triggering cascading failure. Source framing:
"The nodes that receive a huge amount of traffic become hotspots resulting in cascading failure of the nodes."
Mitigations on this wiki¶
- patterns/virtual-nodes-for-load-balancing — smooths distribution-driven hotspots (cause 1).
- patterns/bounded-load-consistent-hashing — caps popularity-driven hotspots (cause 2) by spilling to clockwise fallback nodes.
- patterns/shard-replication-for-hot-keys — copy a hot key to additional shards so reads can spread.
- concepts/hot-row-protection — database-level hotspot mitigation.