Skip to content

CLOUDFLARE

Read original ↗

Introducing Meerkat: an experiment in global consensus

Summary

Cloudflare Research introduces Meerkat, an experimental distributed consensus service built on the QuePaxa algorithm (EPFL, 2023). Meerkat is designed to manage strongly-consistent control-plane state across Cloudflare's 330+ global data centers without the availability penalties of leader-based protocols like Raft. Any replica can drive consensus; there are no required leaders and no timeouts that block progress — making it well-suited for wide-area networks with unpredictable latencies. To Cloudflare's knowledge, this is the first industrial deployment of QuePaxa at global scale.

Key takeaways

  1. Cloudflare needs linearizable control-plane state — placement information, leadership information, and other coordination data must be strongly consistent and accessible from any data center despite crashes, restarts, network failures, and link cuts.

  2. Raft's leader dependency causes real incidents — Cloudflare has experienced multiple production incidents caused by unavailable leaders in Raft-based consensus systems. The leader is a single point of temporary failure, and timeouts are hard to tune on wide-area networks with variable latency.

  3. QuePaxa eliminates the leader bottleneck — any replica can drive consensus for the latest slot. A leader exists for efficiency (1 round-trip vs 3+) but is not required for progress. Concurrent proposals constructively interfere rather than blocking each other (unlike Raft's leadership elections).

  4. Meerkat architecture is log-based — replicas agree on a sequence of log slots via QuePaxa; applications (KV store, leasing) are layered on top by interpreting log events. All replicas maintain the same log (one may lag but never diverges).

  5. Linearizability via log ordering — both reads and writes go through the log. A read proposed to a slot already decided forces the replica to catch up before proposing at the next slot, ensuring no stale reads.

  6. Three QuePaxa advantages over Raft for WAN: (a) no single-replica failure causes unavailability; (b) no leader elections that degrade the system — concurrent proposals cooperate; (c) designed for adversarial asynchronous networks — maintains ~10× higher throughput than Raft/Multi-Paxos under targeted attack conditions per the original paper.

  7. Latency is proportional to inter-replica distance — fundamental to all consensus algorithms. Meerkat offers developer-controlled replica placement, write batching, stale (but never inconsistent) reads, and multi-operation transactions as optimizations.

  8. Not a general-purpose database — Meerkat targets small pieces of control-plane state (leadership for replicated databases, placement information) that are written infrequently but must remain consistent. Consensus round-trip cost makes it unsuitable for high-write-throughput workloads.

  9. Proof-of-concept validated at 50 replicas globally — leaders constantly fail in the test clusters with no increase in error rate. Not yet in production.

  10. Rust implementation, formal verification planned — future posts will cover QuePaxa internals, formal verification of the Rust implementation, bootstrapping/cluster management, optimal replica placement, and deterministic simulation testing.

Operational numbers

Metric Value
Global data centers 330+
Fault tolerance f faults in 2f+1 replicas
Leader round-trips 1 (+ broadcast)
Non-leader round-trips 3+ (+ broadcast)
POC cluster size Up to 50 replicas globally
QuePaxa throughput advantage ~10× vs Raft under adversarial conditions (per paper)

Systems / concepts extracted

Systems

  • systems/meerkat — Cloudflare's new experimental consensus service
  • systems/quepaxa — The QuePaxa consensus algorithm (EPFL, 2023)
  • systems/raft — Referenced as the incumbent consensus algorithm with leader dependency

Concepts

Patterns

Caveats

  • Meerkat is experimental and not yet deployed to production — the post is a groundwork announcement for a series of future posts.
  • No absolute latency or throughput numbers disclosed for the Cloudflare POC; the ~10× claim is from the academic paper's evaluation, not Cloudflare's own benchmarks.
  • QuePaxa's 3+ round-trips for non-leader proposals may still be too slow for latency-sensitive use cases at global scale.
  • Byzantine fault tolerance is explicitly out of scope (same as Raft).
  • The "constantly fail leaders with no error-rate increase" claim lacks published numbers or methodology.

Source

Last updated · 573 distilled / 1,747 read