AWS

Building multi-Region resiliency for AWS CloudFormation custom resource deployment¶

Summary¶

AWS Architecture Blog (2026-07-22) presents a reference architecture for making CloudFormation custom resources resilient across multiple AWS Regions. The core problem: custom resources invoke Lambda functions to perform provisioning logic during stack operations, but CloudFormation provides no native mechanism for multi-Region coordination — no fan-out, no deduplication, no automated failover. The solution is an active-active multi-Region architecture using DynamoDB Global Tables as a distributed lock and idempotency store, SNS cross-region subscriptions for event fan-out, SQS delay queues for secondary-handler timing, and Amazon Application Recovery Controller (ARC) for automated failover detection.

Key takeaways¶

CloudFormation custom resources have no native multi-Region support — each region triggers events independently with no built-in coordination, locking, or failover. Teams either accept single-region risk or build bespoke solutions.
Active-active with single-execution guarantee — both regions are always live and capable of processing, but a DynamoDB Global Table conditional write ensures only one region actually executes the business logic for any given event. This distinguishes the pattern from naive active-active where duplicate side effects are a risk.
DynamoDB conditional writes as distributed lock — the primary handler acquires a lock by writing to the Global Table with a condition expression that only succeeds if no lock exists (attribute_not_exists). This prevents race conditions between the two regions processing the same event.
Delayed secondary as failover mechanism — the secondary SQS queue is configured with a delivery delay (via Delay Queue or Visibility Timeout). After the delay, the secondary Lambda checks DynamoDB: if the primary already processed the event, it skips; if not, it acquires the lock and takes over. The delay window gives the primary first-mover advantage under normal conditions.
SNS cross-region subscriptions for fan-out — customer regions publish lifecycle events to a local SNS topic configured with subscriptions to SQS queues in both infrastructure regions (us-east-1, us-west-2). This is the mechanism that makes the architecture active-active rather than active-passive.
Idempotency records in DynamoDB — beyond locking, the Global Table tracks full request lifecycle state (lock holder, completion status). This means retries, network partitions, and failover scenarios are safe — duplicate processing is prevented at the data layer, not just the application layer.
Amazon Application Recovery Controller for automated failover — CloudWatch alarms monitor SQS queue depth and Lambda execution health. If the primary region degrades, ARC triggers automated failover to the secondary without manual intervention.
Presigned URL callback model — CloudFormation generates a presigned S3 URL with each lifecycle event; the handler must POST success/failure to this URL. Only one handler needs to respond, and the architecture guarantees exactly one does.

Architecture¶

Customer Regions (N)          Infrastructure Regions (2)
┌─────────────────┐          ┌──────────────────────────────────────┐
│ CFN Stack → SNS │──────────│ us-east-1: SQS → Lambda (immediate)  │
│  (lifecycle     │    ╲     │   ↕ DynamoDB Global Table (lock)     │
│   event)        │     ╲    │                                       │
└─────────────────┘      ╲   │ us-west-2: SQS [delay] → Lambda      │
                          ╲──│   ↕ DynamoDB Global Table (replica)   │
                              │                                       │
                              │ ARC: CloudWatch → automated failover  │
                              └──────────────────────────────────────┘

Systems and concepts extracted¶

Systems¶

systems/cloudformation — the orchestrator invoking custom resources
systems/dynamodb — Global Tables used as distributed lock + idempotency store
systems/aws-lambda — custom resource handler execution
systems/aws-sns — cross-region event fan-out
systems/aws-sqs — queue with delay for secondary handler timing
systems/amazon-application-recovery-controller — automated failover orchestration
systems/amazon-cloudwatch — alarm-driven health monitoring

Concepts¶

concepts/distributed-locking — ensuring only one processor handles a given event
concepts/idempotency-token — request-level deduplication across retries/failover
concepts/disaster-recovery-tiers — this architecture sits at the active-active tier
concepts/conditional-writes — the DynamoDB primitive enabling the lock

Patterns¶

patterns/delayed-secondary-with-lock-check — new: queue delay + lock check to give primary priority while enabling seamless failover
patterns/sns-cross-region-fan-out — new: SNS subscriptions targeting SQS in multiple regions for active-active event distribution
patterns/active-active-with-single-execution-guarantee — new: both regions live, but distributed lock ensures exactly-once processing

Operational numbers¶

Regions: architecture tested with us-east-1 (primary) + us-west-2 (secondary)
Customer fan-out: supports N customer regions simultaneously (example: us-east-1, eu-west-1, ap-southeast-1)
Recovery: automated via ARC — no manual intervention for regional failures

Caveats¶

DynamoDB Global Tables use eventual consistency for cross-region replication; the delay queue window must exceed the replication lag to avoid both handlers acquiring the lock during a partition.
The architecture assumes the presigned URL doesn't expire before the secondary's delay window completes — long delays risk URL timeout.
Cost is non-trivial for low-volume custom resources: you're paying for Global Tables, dual Lambda deployments, ARC, and cross-region data transfer even during steady-state.