AWS Architecture Blog

AWS CloudFormation is the foundational tool of infrastructure-as-code for thousands of organizations running workloads on AWS. But as teams push the boundaries of what CloudFormation can do natively, custom resources have emerged as a powerful extension mechanism that unlocks a broad range of possibilities. Yet, when it comes to building resilient, multi-Region deployments with custom resources, customers quickly discover a gap: there is no built-in multi-Region support. In this post, we will explore that challenge and go over a robust active-active architecture that solves it.

A CloudFormation custom resource allows you to write custom provisioning logic that AWS CloudFormation invokes during stack operations (Create, Update, or Delete). When CloudFormation encounters a custom resource in a template, it sends a lifecycle event to a target, typically an AWS Lambda function through an Amazon Simple Notification Service topic. CloudFormation waits for a response with a presigned URL, and then proceeds or rolls back based on that response.

Customers use Custom Resources for a wide variety of use cases, including:

  • Third-party API integrations to provision resources in external systems (for example, DNS providers, SaaS providers) as part of a CloudFormation stack.
  • Complex initialization logic for seeding databases, generating secrets, or bootstrapping configurations that CloudFormation doesn’t natively support.
  • Cross-account or cross-service orchestration for triggering workflows in other AWS accounts or services during stack lifecycle events.
  • Custom validation and compliance checks enforcing organizational policies before a stack is allowed to complete.
  • Resource types not yet supported natively for bridging the gap until AWS adds first-class support.

In short, Custom Resources turn CloudFormation into a fully extensible orchestration engine instead of only an AWS resource provisioner.

While single-Region deployments can achieve high resilience, multi-Region architectures become essential for organizations that need to meet specific business requirements. These include stringent disaster recovery objectives, data residency mandates, latency-sensitive use cases across geographies, and mission-critical business continuity needs. However, when it comes to CloudFormation Custom Resources, multi-Region design introduces a set of hard problems that CloudFormation does not solve natively:

  • No native fan-out mechanism: CloudFormation stacks in different Regions each trigger their own custom resource events independently. There is no built-in way to coordinate these events across Regions.
  • Duplicate execution risk: If you deploy the same Lambda function handler in multiple Regions to achieve redundancy, both instances may process the same event. This can lead to duplicate side effects (for example, creating the same record twice in a database or calling an external API multiple times).
  • No distributed locking: CloudFormation provides no mechanism to verify that only one handler processes a given event, even when multiple handlers are active.
  • No automated failover: If the primary Region’s Lambda function handler fails, there is no built-in mechanism to automatically route the event to a secondary Region.
  • Idempotency is your problem: Helping verify that retries and failover scenarios don’t cause unintended duplicate operations is entirely the responsibility of the developer.

Until now, these gaps meant that teams either accept the risk of single-Region custom resource handlers (a reliability concern) or build complex, bespoke solutions to handle multi-Region scenarios.

Walkthrough

Prerequisites

Solution approach

This proposed architecture delivers an active-active multi-Region solution for CloudFormation custom resource processing. It is designed around four core principles:

  • Active-Active processing: Both the primary Region (us-east-1) and secondary Region (us-west-2) are always live and capable of handling events.
  • No duplicate execution: A DynamoDB Global Table-based distributed locking mechanism helps verify that only one Region processes any given event, regardless of which Region receives it first.
  • Idempotency mechanism: Every request is tracked by state, so retries and failover scenarios are designed to avoid duplicate side effects.
  • Fully automated failover: Amazon Application Recovery Controller detects failures and triggers failover without manual intervention.

This architecture avoids the single points of failure inherent in single-Region custom resource designs while preventing the duplicate processing risks of naive multi-Region approaches. This architecture is ideal for mission-critical workloads where regional failures cannot be tolerated.

The following section provides a detailed walkthrough of how this architecture processes a CloudFormation lifecycle event from end to end.

This architecture diagram describes a multi-Region CloudFormation custom resource architecture operating in an Active-Active configuration. It handles CloudFormation lifecycle events (Create/Update/Delete) with high availability and no duplicate processing across multiple AWS Regions. The architecture uses a central primary Region (us-east-1) and a secondary Region (us-west-2) to process custom resource events, with Amazon DynamoDB Global Tables providing distributed locking and idempotency, and Amazon Application Recovery Controller providing automated failover. Customer AWS Regions fan out events simultaneously to both infrastructure Regions, supporting resilience even if the primary Region fails.

Architecture diagram showing the multi-Region CloudFormation custom resource processing flow with SNS fan-out, SQS queues, Lambda handlers, DynamoDB Global Tables for distributed locking, and Amazon Application Recovery Controller for automated failover

Step 1: Event initiation (Customer Regions)

A CloudFormation Stack in one of the customer Regions (us-east-1, eu-west-1, or ap-southeast-1) initiates a Create, Update, or Delete lifecycle event. Along with the event payload, CloudFormation generates a presigned response URL that the handler must call to signal success or failure. This event is published to a local Amazon Simple Notification Service (SNS) topic within that customer Region.

Step 2: Cross-Region fan-out with SNS subscriptions

The Amazon SNS topic is configured with cross-region subscriptions that simultaneously fan out the event to two Amazon SQS queues in the central infrastructure AWS Regions:

  • Primary SQS queue: Central Infrastructure Region: us-east-1.
  • Secondary SQS queue: Central Infrastructure Region: us-west-2.

Both queues receive the event at the same time, setting up the Active-Active processing model.

Step 3: Primary Lambda function handler (Immediate processing)

The Primary SQS queue triggers the Primary Lambda Custom Resource Handler immediately, with no delay. The Lambda function executes the following steps:

  • Check the DynamoDB Global Table for an existing lock on this request.
  • Acquire the lock by using a conditional write (only succeeds if no lock exists, preventing race conditions).
  • Execute the custom resource business logic.
  • Send a SUCCESS or FAILED response back to CloudFormation using the presigned URL.
  • Update the DynamoDB state to mark the request as fully processed.

Step 4: Secondary Lambda function handler (Delayed processing)

The Secondary SQS queue is configured with a delay, implemented by using either an SQS Delay Queue or a Visibility Timeout. After this delay, the Secondary Lambda Custom Resource Handler runs:

  • Check the DynamoDB Global Table Replica for an existing lock.
  • Skip processing if the primary has already handled the request (idempotency check).
  • Acquire the lock if the primary has not yet processed it (failover scenario).
  • Execute the custom resource logic if the lock was successfully acquired.
  • Send the response to CloudFormation.

The delay is intended to give the primary Region time to process the event first. The secondary only takes over if the primary has not completed processing within the delay window.

Step 5: DynamoDB Global Tables: Distributed locking and idempotency

Amazon DynamoDB Global Tables are the backbone of coordination in this architecture. Both Regions read from and write to the Global Table with strong consistency. The table tracks:

  • Lock state: Which Region holds the lock for a given request.
  • Idempotency records: Whether a request has already been processed.
  • Request state: The full lifecycle status of each event.

Bidirectional replication is designed to help maintain both Regions with the latest state, supporting the lock mechanism’s reliability despite network partitions or regional degradation.

Step 6: CloudFormation response

After either the primary or secondary Lambda function handler completes processing, CloudFormation receives the SUCCESS or FAILED callback using the pre-signed URL. Based on this response, CloudFormation either continues the stack operation or initiates a rollback.

Step 7: Amazon CloudWatch monitoring

Amazon CloudWatch alarms continuously monitor SQS queue depth and Lambda execution health in both Regions. These alarms serve as the early warning system for the automated failover mechanism.

Step 8: Automated failover with ARC

If the primary Region (us-east-1) experiences a failure, CloudWatch detects it and triggers Amazon Application Recovery Controller to initiate automated failover to the secondary Region (us-west-2). No manual intervention is typically required. The secondary Region is designed to take over processing responsibilities.

Clean up

Delete resources created using the following AWS services in every Region to avoid additional costs:

  • ARC.
  • SNS topic.
  • SQS queue/messages.
  • DynamoDB table.
  • Lambda code.
  • CloudWatch Log Groups.
  • IAM roles.
  • CloudFormation (if used for automation).
  • Any other AWS service used for customizing your deployment.

Conclusion

CloudFormation Custom Resources are an indispensable tool for teams building sophisticated infrastructure automation on AWS. However, the lack of native multi-Region support has long been a barrier to building truly resilient custom resource architectures.

This architecture addresses the major challenge directly:

  • Resilience: Active-Active design means no single Region is a bottleneck or single point of failure.
  • Correctness: Amazon DynamoDB distributed locking and idempotency designed to eliminate duplicate processing.
  • Automation: Amazon Application Recovery Controller-driven failover removes the need for manual intervention during regional outages.
  • Scalability: The fan-out model with SNS cross-Region subscriptions supports multiple customer Regions simultaneously.

For teams operating at scale across multiple AWS Regions, this architecture provides a blueprint for extending the power of CloudFormation without sacrificing reliability. Whether you’re managing compliance-driven multi-Region deployments or building for global high availability, this pattern gives you a foundation for resilient custom resource processing.

Call to action

Refer to the following content to learn more about relevant AWS services: