Skip to content

AWS 2026-10-14

Read original ↗

Multi-region disaster recovery for Amazon API Gateway WebSocket APIs

Summary

This AWS Architecture Blog post presents a design pattern for active-passive multi-Region disaster recovery of real-time applications built on Amazon API Gateway WebSocket APIs. WebSocket connections are persistent and bound to a single Region, so a Regional outage severs every active connection and loses the per-session state built on top of them. The pattern's core idea is to separate application state from the connection itself: identical WebSocket stacks are deployed in a primary and secondary Region, shared state is replicated via DynamoDB Global Tables, Route 53 failover routing fronts both behind one custom domain, a canary health probe drives automatic DNS failover, and client-side reconnection logic (heartbeat + exponential backoff with full jitter + a getState resync) restores the session transparently in the surviving Region. The demo is a real-time auction where the current bid survives a simulated Regional failure. (Source: article)

Key takeaways

  • Persistent connections are the DR hard part. A WebSocket connection is bound to one Region; a Regional failure severs every active connection at once. The design's premise is to decouple session state from the socket so a reconnect in another Region can resume the session, rather than trying to migrate the connection itself. (Source: article)
  • Active-passive topology with a warm secondary. Identical stacks run in primary (us-east-1) and secondary (us-west-2); the secondary is kept continuously ready by state replication rather than created on failover — a warm/pilot-light-style standby that pairs with async cross-region replication. See concepts/regional-failover (active-passive).
  • State lives in DynamoDB Global Tables, keyed by user, not connection. DynamoDB Global Tables replicate application state to the secondary Region asynchronously, typically within ~1 second. Production guidance: key stored state by user ID (e.g. the sub claim), not by the connection ID, because the connection ID changes on every reconnect. (Source: article)
  • DNS failover is driven by a traffic-independent canary, not live traffic. An EventBridge rule fires a canary Lambda every minute; the canary opens a WebSocket, sends a heartbeat, and publishes a custom CloudWatch metric (App/WebSocketHealth, dimension ApiId) — 1 healthy / 0 unhealthy. A CloudWatch alarm with TreatMissingData: breaching fires after two consecutive unhealthy datapoints, and the associated Route 53 health check (InsufficientDataHealthStatus: Unhealthy) marks the primary down, shifting DNS to the secondary. (Source: article)
  • Asymmetric thresholds prevent failover flapping. Recommended for production: a low threshold (2 datapoints) for fast failover and a higher threshold (>5) for slower failback, so an unstable primary recovering doesn't flap traffic back and forth. DNS TTL 60–120 s balances propagation speed against lookup efficiency. (Source: article)
  • Failback is automatic through the same mechanism. When the primary recovers the canary succeeds again, the alarm clears, and Route 53 shifts DNS back — no manual intervention. (Source: article)
  • Clients detect breakage via heartbeat and reconnect with backoff + jitter. Both client and canary use an application-level heartbeat per RFC 6455: a lightweight message expecting an ack (pong). The client force-closes and reconnects after 2 consecutive failed heartbeats; with a 30 s interval that detects a degraded (not just unreachable) backend within ~60 s. Reconnection uses exponential backoff with full jitter to avoid a thundering herd slamming the secondary the moment DNS shifts. See concepts/exponential-backoff-jitter. (Source: article)
  • Re-auth and state resync on reconnect. WebSocket APIs authenticate on the $connect route, so a Lambda authorizer must be deployed in both Regions; after reconnect the client sends a getState message to recover the latest state from the local DynamoDB replica. (Source: article)
  • Degraded ≠ unreachable. API Gateway returns errors when the backing Lambda is throttled, so the socket stays technically open while the app is non-functional — tracking consecutive bad heartbeat responses (not just socket close) is what catches this grey-failure case. (Source: article)

Architecture (normal operation → failover)

  1. Client opens a WebSocket to the app's custom domain.
  2. Route 53 failover routing policy resolves to the primary Region under normal conditions.
  3. API Gateway (primary) invokes a Lambda per connection event / message.
  4. Lambda reads/writes app state in DynamoDB; connected clients receive updates over their open sockets.
  5. DynamoDB Global Tables replicate state to the secondary Region asynchronously (~1 s).
  6. Independently, an EventBridge rule triggers the canary Lambda every minute.
  7. Canary opens a socket, sends a heartbeat, publishes the result as a custom CloudWatch metric.
  8. A CloudWatch alarm evaluates the metric; two consecutive failures (or missing data) → alarm → Route 53 health check marks primary unhealthy.
  9. Route 53 begins resolving the domain to the secondary Region.
  10. Clients reconnect through the same domain, now reach the secondary, and recover state from the local DynamoDB replica.

Operational numbers

  • State replication lag: DynamoDB Global Tables ~1 second (asynchronous); writes immediately before a failure may not reach the secondary — a small but non-zero RPO window.
  • Failover detection: ~2 minutes (two consecutive failed canary probes at 1-minute intervals).
  • Canary cadence: every 1 minute (EventBridge rule).
  • Client heartbeat: interval 30 s, timeout 35 s; force-close after 2 consecutive failed heartbeats → degraded-backend detection within ~60 s.
  • Reconnect backoff: min(2000 × 2^attempt, 20000) ms cap, randomized with full jitter (Math.random() × expDelay).
  • Alarm thresholds (recommended): fast failover at 2 datapoints; slower failback at >5 datapoints to prevent flapping.
  • DNS TTL: 60–120 s.
  • Relational alternative: Amazon Aurora Global Database gives sub-second cross-Region replication but requires promoting a read replica to primary on failover, adding ~1–2 minutes to recovery (can be automated off a CloudWatch alarm).

Caveats

  • Non-zero RPO by construction. Asynchronous Global Tables replication means writes in the last ~1 s before failure can be lost; the post calls this window "small enough to be acceptable" for most apps but it is a real data-loss window. (Source: article)
  • Illustrative, not a scaled production writeup. Numbers are design/demo values (auction sample app), not measured production percentiles or throughput. The reconnection snippets use the react-use-websocket library; the patterns are library-agnostic.
  • Lambda authorizer must exist in both Regions or reconnecting clients can't re-authenticate against the secondary — easy to miss when only the primary is fully exercised in testing.
  • Session modeling is app-specific. Keying by user ID vs connection ID, and what counts as "session state," is left to the implementer.
  • The GitHub sample (aws-samples/sample-apigw-websocket-multi-region-disaster-recovery) ships CloudFormation templates + a React client; manual cleanup of Route 53 failover records and custom domain names is required.

Systems / concepts / patterns extracted

Source

Last updated · 854 distilled / 2,448 read