Skip to content

PATTERN Cited by 1 source

Failover cluster escape hatch

Definition

When a managed service's control plane becomes unavailable during an incident (blocking scaling, reboots, or reassignment), recovery can stall indefinitely. The failover-cluster escape hatch pre-provisions a clean parallel cluster using infrastructure-as-code templates so operators can route producers and consumers to an entirely fresh environment without depending on the degraded cluster's management APIs.

How it works (Atlassian StreamHub)

A four-stage runbook:

  1. Shard failover (~15 min): Route traffic from the affected shard to a healthy shard; apply rate limits and quarantine to stay within target capacity.
  2. Prepare (~2 hours): Provision a new MSK cluster via IaC templates (topics, tiered storage config, ACLs, client config).
  3. Evaluate: Continue mitigating the primary cluster while the clean environment initializes.
  4. Cluster failover: If the primary remains stuck past a decision threshold, update routing so all producers/consumers move to the clean cluster.

Why this pattern exists

Managed streaming services abstract away operations but introduce a hidden dependency: the control plane. When the control plane is unhealthy, you cannot scale brokers, expand storage, rebalance partitions, or complete automated recovery — precisely the levers needed during incidents. The escape hatch ensures recovery does not depend entirely on the failing control plane becoming healthy.

Seen in

Last updated · 602 distilled / 1,824 read