Skip to content

SYSTEM Cited by 1 source

Meerkat

Meerkat is Cloudflare Research's experimental distributed consensus service, powered by the QuePaxa algorithm. Designed to manage strongly-consistent control-plane state across Cloudflare's 330+ global data centers without the availability penalties of leader-based protocols like Raft.

Architecture

Developers request a cluster of Meerkat replicas, each connected to every other replica. Each replica participates in consensus and can receive both reads and writes. Developers specify which data centers host their replicas; Meerkat places them automatically.

Under the hood, replicas maintain a replicated log — a sequence of decided slots. Applications (key-value store, distributed leases) are layered on top by interpreting log events. All functioning replicas maintain the same log; one may lag behind but will never record different entries.

Key properties

  • Linearizability — reads and writes are ordered exactly as they occurred in real time
  • Leaderless — any replica can drive consensus for the latest slot
  • Fault-tolerant — available for reads and writes as long as a majority of replicas (more than half) are alive and can communicate
  • No timeouts blocking progress — unlike Raft, progress is never halted due to a timeout

Applications

  • Transactional key-value store (compare-and-swap, general transactions)
  • Distributed leasing system
  • Control-plane coordination (placement information, leadership information)

Why not Raft

Cloudflare experienced multiple production incidents from unavailable leaders in Raft-based systems. Two structural problems with authoritative leaders on wide-area networks:

  1. Leader failure → system unavailability until a new leader is elected via timeouts
  2. Timeout tuning is intractable on networks with unpredictable latencies — too short means constant false elections; too long means slow recovery; concurrent campaigns interfere destructively

Performance characteristics

  • Leader proposals: 1 round-trip + broadcast notification
  • Non-leader proposals: 3+ round-trips + broadcast
  • Write batching amortizes consensus cost
  • Stale (but never inconsistent) reads available without consensus rounds
  • Multi-operation transactions supported (single consensus round for multiple ops)

Status

Experimental; not yet deployed to production. Proof-of-concept clusters validated with up to 50 globally-distributed replicas. Leaders constantly fail with no increase in error rate.

Implementation

Written in Rust. Formal verification of parts of the implementation planned. Deterministic simulation testing used for bug finding.

Future work (announced)

  • Full QuePaxa algorithm explanation
  • Formal verification approach
  • Bootstrapping and cluster management
  • Optimal replica placement algorithms
  • Deterministic simulation testing methodology
  • Peer-reviewed manuscript

Seen in

Last updated · 573 distilled / 1,747 read