Skip to content

CONCEPT Cited by 1 source

Failure-tolerance envelope

Definition

The failure-tolerance envelope is the operator-specified bound on the number and kind of concurrent failures a system is engineered to survive without data loss. Within the envelope, the system continues serving requests correctly. Outside the envelope, the system is forced to choose between stalling (preserving correctness) and abandoning unreachable state (preserving availability) — the availability-vs-data-loss trade-off.

The envelope is a design input, not a protocol constant. Different deployments have different envelopes; the consensus protocol's job is to encode the envelope as a durability predicate and derive the matching election / read-quorum reach from it via intersecting quorums.

Canonical example

Sugu Sougoumarane's Part 3 treatment of the 5-node 2-durability case:

"In the above five node case, if two nodes fail, the failure tolerance has been exceeded. We can only reach three nodes. If we don't know about the state of the other two nodes, we will have to assume the worst case scenario that a durable request could have been accepted by the two unreachable nodes. This will cause the election process to stall." (Source: sources/2026-04-21-planetscale-consensus-algorithms-at-scale-part-3-use-cases)

The envelope here is "no more than one simultaneous node failure". A single failure is survivable: the other 4 nodes intersect with every possible 2-durability write set. Two failures exceed the envelope: only 3 nodes remain reachable, and an election cannot guarantee it touches a node that holds the last durable request.

How the envelope composes with the durability predicate

Envelope Durability predicate Election reach Notes
Up to 1 node failure on N=5 2-of-5 4-of-5 Sugu's worked example.
Up to 1 zone failure, 3 zones ≥ 1 per ≥ 2 zones Cover all 3 zones Multi-zone durability.
Up to 1 region failure, 2 regions ≥ 1 per region Both regions Proactive transfer on region maintenance.
1 AZ + 1 segment, 3 AZ × 2 segments 4 of 6 3 of 6 Aurora's geometry.

The envelope is the operator's statement of intent; the protocol's job is to encode it safely.

Worked examples from Part 3 (all four canonical shapes)

  1. Fifteen-primary, three-DC — envelope: "We don't expect two nodes to go down at the same time. Network partitions can happen, but only between two data centers; a data center will never be totally isolated. A data center can be taken down for planned maintenance."
  2. Four-zone, one-node-per-zone — envelope: "Any node can fail. A zone can go down without notice. A partition can happen between any two zones."
  3. Six-node, three-zone — envelope: same as #2 structurally; more nodes per zone give richer quorum flexibility.
  4. Two-region, two-zones-per-region — envelope: "We don't expect more than one zone to go down. A region can be taken down for maintenance, in which case we want to proactively transfer writes to the other region."

Each envelope statement maps to a different durability predicate. Each predicate implies a different election reach. None map cleanly to majority quorum. Part 3's central observation is that the fixed-majority assumption forces operators to either mis-specify the envelope or run the protocol against a mismatched one.

When failures exceed the envelope

Sugu: "If this were to happen, the system has to allow for a compromise: abandon the two nodes and move forward. Otherwise, the loss of availability may become more expensive than the potential loss of that data."

This is the availability-vs-data-loss trade-off in operational form. The protocol cannot make this decision — the operator does. The protocol's responsibility is to detect the out-of-envelope condition and surface the choice rather than silently choosing one option.

Concrete shapes of the compromise:

  • Stall: refuse to elect a new leader until the unreachable nodes rejoin. Preserves all previously-ack'd writes. Costs availability for the duration.
  • Abandon: elect a new leader from the reachable subset, accept that any writes on the unreachable side that weren't seen by the reachable side are lost. Preserves availability. Costs durability.
  • Partial: human decision. Often used for the "cut the network cable" / "dispatch a human" emergency scenarios Sugu cites from Google.

Envelope tightening via cross-boundary predicates

A subtle lesson of Part 3: you can tighten the envelope without raising the durability node count, by changing the predicate from "any k nodes" to "k nodes across network boundaries".

"This type of failure can happen if the leader and the recipient node are network partitioned from the rest of the cluster. We can mitigate this failure by requiring the ackers to live across network boundaries. The likelihood of a replica node in one cell failing after an acknowledgment, and a master node failing in the other cell after returning success, is much lower."

Same k; tighter envelope (now resistant to single-cell partition + crash correlation). The predicate encodes operational knowledge about failure correlation — something majority-quorum systems cannot express.

Correlated failure breaks envelope assumptions

The envelope is only as good as the independent-failure assumption it rests on. See concepts/correlated-ebs-failure for the fleet-level empirical case: an envelope stated as "independent EBS failures, 1% per month" is meaningless if a regional event takes down all replicas simultaneously. Real deployments pay down correlated-failure risk by:

  • Across-zone / across-region replicas — geographic independence.
  • Heterogeneous hardware / firmware — batch independence.
  • Staggered software rollouts — version independence.
  • Time-diversified backups — temporal independence.

Any of these are envelope-tightening mechanisms that operate orthogonally to the durability predicate's quorum count.

Why the envelope belongs at the protocol boundary

Majority-quorum protocols encode a specific envelope — "up to (N-1)/2 simultaneous failures" — into the protocol mechanics. This works well for homogeneous, symmetric deployments but fails whenever the deployment's actual envelope differs:

  • A "no more than one zone at a time" envelope on N=6 nodes across 3 zones is not 3-of-6 — it's "at least 1 per zone across ≥ 2 zones".
  • A "cross-region maintenance with proactive transfer" envelope isn't a symmetric quorum at all — it's "either region can be the sole writer while the other is under maintenance."

Part 3's architectural conclusion: the envelope is the input to the protocol, not a consequence of protocol choice. Separating the two means the operator can encode their failure-tolerance reasoning directly, and the protocol treats it as configuration.

Seen in

Last updated · 550 distilled / 1,221 read