Skip to content

CONCEPT Cited by 1 source

Split-brain

Definition

Split-brain is the failure mode in which two (or more) nodes simultaneously believe they are the sole owner / authority for a resource that is supposed to have exactly one owner — and each acts on that belief independently. For a single-leader replicated system this is two primaries both accepting writes; for a catalog it is two catalogs both coordinating commits for the same table; for a gossip cluster it is two sub-partitions each treating itself as the whole. The defining danger is that no component throws an error — the system is internally consistent from inside each half, so the divergence is silent until the halves try to reconcile.

The result is almost always data divergence and silent data loss: the two owners accept conflicting mutations, forking what was meant to be a single source of truth. Recovery usually requires discarding or manually merging one branch.

Why it happens

Split-brain arises whenever the mechanism that is supposed to enforce single-ownership can be defeated by independence or partition:

  • Network partition. Each side of a partition can no longer see the other and (if not fenced) promotes itself. Sub-partitions "keep gossiping happily with each other; healing exposes the divergence — split-brain is invisible from inside each half" (Source: concepts/gossip-protocol).
  • Independent authorities that don't communicate. When two systems that each claim ownership have no channel to coordinate, nothing detects the dual claim. Databricks' framing for Iceberg catalogs: "catalogs are independent and do not communicate," so attaching a table to a second catalog while it is still active in the first leaves both believing they are the sole owner — "neither will throw an error, but the first write operation will fork the table, leading to inconsistent query results and silent data loss" (Source: sources/2026-10-05-databricks-unlocking-data-portability-preventing-catalog-lock-in-with-register-and-unregister).
  • Non-atomic ownership transfer. Establishing a new owner before the old owner has relinquished control leaves a window where both satisfy the ownership invariant.

Prevention mechanisms

  • Singleton-writer enforcement. "A system-of-record writer must be singleton to avoid split-brain" (concepts/leader-election) — a consensus/lease layer ensures at most one writer at a time.
  • Fencing. Explicitly invalidate the old owner (reject its stale epoch/proposal number, stop replicas accepting its writes) before the new owner is active.
  • Revoke-before-establish ordering. Make the old owner explicitly relinquish control, and only then let the new owner take over, so the single-owner invariant is never doubly satisfied (patterns/separate-revoke-from-establish). The Iceberg UNREGISTER → REGISTER handoff is a concrete instance: UNREGISTER makes the old catalog forget the table and emit the metadata pointer; REGISTER on the new catalog only then assumes commit coordination, guaranteeing "exactly one managing catalog at all times."

Why it is hard to detect

The hallmark of split-brain is silence: each owner is locally correct and raises no error. Divergence surfaces only when the two forks are read side-by-side or an attempt is made to reconcile them — by which point conflicting writes may already be durable. This is why prevention (fencing / singleton enforcement / ordered handoff) is strongly preferred over detection-and-repair; conflict-resolution strategies like concepts/last-write-wins can bound the damage but cannot recover lost intent.

Seen in

Last updated · 854 distilled / 2,448 read