Skip to content

PATTERN Cited by 1 source

CRDT over Raft for WAN state distribution

Problem

You need every node in a globally-distributed fleet (dozens of regions, thousands of servers) to converge on the same view of some dataset. Every node runs the same decision logic off that view. Updates flow from each node (each node is the source of truth for its own slice). You have to pick a state-distribution substrate.

The candidate families:

  • Consensus — Raft, Paxos, etcd, Consul, ZooKeeper, rqlite, FoundationDB. Strong consistency, linearizable writes, quorum commit.
  • Centralized store — a single big database (FoundationDB, S3-backed stores). Strong consistency; you can buy your way out of most problems.
  • Gossip + CRDT — SWIM + QUIC + CRDT-SQLite (Corrosion's shape). Eventual consistency; no central coordination; conflict-free merge.

Pattern

Pick gossip + CRDT when:

  1. The deployment is global-WAN — RTTs make consensus protocols pay a painful tax on every write.
  2. Writes rarely conflict — each node owns its own slice of the keyspace; the common case doesn't need a merge rule.
  3. Reads are local and eventually-consistent is fine — consumers can tolerate a few seconds of staleness while updates propagate.
  4. The workload doesn't need global invariants — no unique constraints, no global referential integrity, no cross-shard transactions.

If those conditions hold, gossip + CRDT wins on:

  • Latency: reads are local SQLite queries, milliseconds.
  • Throughput: no quorum bottleneck; every node accepts writes.
  • Availability: no leader election to fail; partitions heal by gossip.
  • Operational surface: no quorum reconfiguration, no failover runbooks.

Consensus still wins when you need linearizability, global invariants (uniqueness, referential integrity), or small-tight-coupled-cluster coordination (service discovery within a bounded fleet, leader elections, lock services). The advice isn't "never use Raft" — it's "don't use Raft across a WAN."

Fly.io's framing

From sources/2025-10-22-flyio-corrosion:

"Like an unattended turkey deep frying on the patio, truly global distributed consensus promises deliciousness while yielding only immolation. Consensus protocols like Raft break down over long distances. And they work against the architecture of our platform: our Consul cluster, running on the biggest iron we could buy, wasted time guaranteeing consensus for updates that couldn't conflict in the first place."

Fly ran Consul as their global routing store. It ran on their biggest iron. It spent its cycles on consensus for updates that couldn't conflict. They replaced it with Corrosion — gossip + CRDT — and the shape matched their orchestration model better.

Design rule of thumb

  • Intra-region: consensus is fine. LAN RTTs keep the cost manageable; linearizability is cheap.
  • Inter-region: gossip + CRDT. Accept eventual consistency; use CRDTs for convergence; bound blast radius with regionalization.

Caveats

  • CRDT design is an expertise — bad CRDT schemas leak all the way to your operational hazards ( nullable-column backfill amplification is a case in point).
  • "Writes rarely conflict" needs to be a designed property, not a hope. Enforce it structurally (one-writer-per-key).
  • Blast radius is different — a gossip bug propagates globally in seconds; a consensus bug might wedge the cluster but not propagate corruption.
  • Regionalization is usually the right companion pattern once the gossip cluster grows — see patterns/two-level-regional-global-state.

Seen in

Last updated · 200 distilled / 1,178 read