Skip to content

CONCEPT Cited by 1 source

Leader timeout problem

The leader timeout problem is the fundamental difficulty of configuring failure-detection timeouts in leader-based consensus protocols (e.g., Raft) on networks with unpredictable latencies.

The dilemma

When a leader fails, non-leader replicas must detect the failure via a timeout — if they haven't heard from the leader in some configured duration, they propose themselves as the new leader. Three failure modes emerge:

  1. Timeout too short — replicas time out during normal high-latency periods, triggering unnecessary leader elections that block all writes during the transition
  2. Timeout too long — the system reacts slowly to an actual leader failure, during which all writes are blocked
  3. Simultaneous campaigns — multiple replicas time out and propose themselves simultaneously, causing their campaigns to interfere with each other and perpetuating the write-blocking state

Why it's worse on WANs

On wide-area networks (e.g., Cloudflare's 330+ data centers across the Internet), latencies vary wildly and unpredictably. There is no single timeout value that works well across all network conditions at all times. Cloudflare has experienced multiple real production incidents caused by this exact failure mode in Raft-based systems.

Solutions

  • Leaderless consensus (concepts/leaderless-consensus) — eliminate the required leader entirely (e.g., QuePaxa / Meerkat)
  • Adaptive timeouts — dynamically adjust based on observed latency (partial mitigation; doesn't fix the fundamental trade-off)
  • Pre-vote (Raft extension) — reduces disruptive elections but doesn't eliminate them

Seen in

Last updated · 573 distilled / 1,747 read