Definition
- UPDATE_ROLLBACK_FAILED – a terminal CloudFormation stack state where an update failed and the automatic rollback also failed, leaving the stack stuck.
- Root causes – the resource was manually modified outside CloudFormation (drift), the resource is unreachable or stuck, or the deployment role lacks permissions for the rollback.
- Recovery – diagnose from stack events, fix the cause out of band, then call
continue-update-rollback. - Skip – if a resource cannot be reconciled, pass
--resources-to-skipwith its logical ID; CloudFormation skips that resource and completes the rest. - Post-skip – after skipping, CloudFormation's record is out of sync; reconcile via resource import or a subsequent change set.
Imagine National Bank of India is renovating a branch using a strict blueprint system. You submit a change: 'move the vault and add a counter.' Halfway through, the vault installation fails. The system tries to restore the original layout, but discovers someone had bricked over the old vault doorway without telling the system. Now it is stuck: it cannot finish the renovation and it cannot restore the original. To get unstuck, a human fixes the doorway (fix the resource) or tells the system to skip the vault (resources-to-skip), then says 'continue putting the rest back.'
Understanding the Concepts
What is UPDATE_ROLLBACK_FAILED?
CloudFormation manages infrastructure as a stack and keeps an internal record of every resource's expected state. When you submit an update and a resource fails, CloudFormation automatically begins a rollback. If the rollback itself cannot finish, the stack enters UPDATE_ROLLBACK_FAILED. You cannot run a new update until this is resolved.
Stack state transitions:
UPDATE_IN_PROGRESS
│
▼ (resource fails)
UPDATE_ROLLBACK_IN_PROGRESS
│
├──▶ (rollback succeeds) ──▶ UPDATE_ROLLBACK_COMPLETE (safe, try again)
│
└──▶ (rollback fails) ──▶ UPDATE_ROLLBACK_FAILED (stuck!)
What are the three common causes?
The three root causes are drift, unreachable resources, and insufficient permissions.
Cause 1: Drift ──────────────── Someone modified a resource manually in the console. CloudFormation tries to restore its recorded version. The live state no longer matches. Operation fails. Example: on-call engineer hand-edited a security group rule during an incident. CloudFormation's record says the rule should be X, but it is now Y. Cause 2: Unreachable resource ────────────────────────────── An EC2 instance that will not stop. An Auto Scaling group that cannot reach capacity. A Lambda custom resource that keeps failing its response. Cause 3: Insufficient permissions ────────────────────────────────── The IAM role CloudFormation uses lacks a permission needed for the rollback action, such as deleting a resource it created.
# Diagnose: read stack events to find the failing resource
aws cloudformation describe-stack-events \
--stack-name bank-payments-stack \
--query 'StackEvents[?ResourceStatus==`UPDATE_FAILED`].[LogicalResourceId,ResourceStatusReason]' \
--output table
# Example output: ┌──────────────────────┬──────────────────────────────────────────────────┐ │ LogicalResourceId │ ResourceStatusReason │ ├──────────────────────┼──────────────────────────────────────────────────┤ │ BankSecurityGroup │ Resource sg-0abc was modified outside of stack │ │ ArchiveBucket │ Bucket was deleted outside of CloudFormation │ └──────────────────────┴──────────────────────────────────────────────────┘
How do you recover with continue-update-rollback?
First fix the underlying cause out of band, then tell CloudFormation to retry the rollback.
# Step 1: Fix the drifted security group to match CloudFormation's record
aws ec2 revoke-security-group-ingress --group-id sg-0abc --protocol tcp --port 8080 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-id sg-0abc --protocol tcp --port 443 --cidr 10.0.0.0/8
# Step 2: Add missing permission if that was the cause
aws iam attach-role-policy \
--role-name bank-cfn-deploy-role \
--policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
# Step 3: Continue the rollback
aws cloudformation continue-update-rollback --stack-name bank-payments-stack
How do you skip resources that cannot be reconciled?
If a resource was deleted outside CloudFormation and cannot be recreated during rollback, skip it.
# Skip specific resources that cannot be reconciled
aws cloudformation continue-update-rollback \
--stack-name bank-payments-stack \
--resources-to-skip ArchiveBucket
# CloudFormation skips ArchiveBucket, completes rollback for everything else
# Stack moves to UPDATE_ROLLBACK_COMPLETE
package com.bank.ops;
import software.amazon.awssdk.services.cloudformation.CloudFormationClient;
import software.amazon.awssdk.services.cloudformation.model.ContinueUpdateRollbackRequest;
import java.util.List;
public class StackRecovery {
private final CloudFormationClient cfn;
public StackRecovery(CloudFormationClient cfn) {
this.cfn = cfn;
}
public void recoverStack(String stackName, List<String> resourcesToSkip) {
ContinueUpdateRollbackRequest request = ContinueUpdateRollbackRequest.builder()
.stackName(stackName)
.resourcesToSkip(resourcesToSkip)
.build();
cfn.continueUpdateRollback(request);
System.out.println("Rollback continued for " + stackName
+ ", skipping: " + resourcesToSkip);
}
}
What happens after skipping a resource?
After a skip, CloudFormation's record of that resource is out of sync with reality. You must reconcile it afterward.
Post-skip reconciliation options:
1. Import the resource into the stack
aws cloudformation import ... --resources-to-import ...
2. Remove the resource from the template and run an update
(CloudFormation stops tracking it)
3. Recreate the resource and import it
(if the resource was deleted)
If you do nothing, the next update may fail the same way.
How do you prevent UPDATE_ROLLBACK_FAILED?
Prevention is far better than recovery.
# 1. Forbid manual console edits to CloudFormation-managed resources
# Use IAM policies that deny direct modification of tagged resources
# 2. Run drift detection before large updates
aws cloudformation detect-stack-drift --stack-name bank-payments-stack
aws cloudformation describe-stack-drift-detection-status \
--stack-drift-detection-id <id>
# 3. Use an explicit least-privilege service role
aws cloudformation update-stack \
--stack-name bank-payments-stack \
--template-body file://template.yaml \
--role-arn arn:aws:iam::123456789012:role/bank-cfn-deploy-role
# 4. Enable termination protection
aws cloudformation update-termination-protection \
--enable-termination-protection \
--stack-name bank-payments-stack
# 5. Use change sets to review before applying
aws cloudformation create-change-set \
--stack-name bank-payments-stack \
--template-body file://template.yaml \
--change-set-name review-changes
How do nested stacks affect recovery?
A parent stack can be stuck because a child stack is stuck. You must resolve the child first.
Nested stack recovery order:
Parent: bank-payments-stack (UPDATE_ROLLBACK_FAILED)
└── Child: bank-payments-db-stack (UPDATE_ROLLBACK_FAILED)
└── Cause: RDS instance was manually modified
Fix order:
1. Fix the RDS instance in the child stack
2. continue-update-rollback on the child stack
3. continue-update-rollback on the parent stack
What are common mistakes with CloudFormation rollback?
# WRONG: skipping resources without reconciling afterward
aws cloudformation continue-update-rollback \
--stack-name bank-payments-stack \
--resources-to-skip BankSecurityGroup
# The security group is now out of sync. Next update may fail again.
# CORRECT: skip, then reconcile in a follow-up change set
# After rollback completes, import the resource or update the template
# WRONG: hand-editing CloudFormation-managed resources in the console
"I'll just quickly fix this security group rule in the console..."
# This is the #1 cause of UPDATE_ROLLBACK_FAILED
# CORRECT: all changes go through CloudFormation templates
# If an emergency fix is needed, document it and reconcile immediately
What should a senior engineer know?
In a bank you cannot simply tear down and recreate a production stack holding live account data. The senior discipline is prevention: forbid manual edits, run drift detection before updates, use explicit least-privilege service roles, and review changes via change sets. When recovery is needed, the process is deliberate: diagnose from stack events, fix the cause out of band, continue the rollback, and reconcile any skipped resources afterward.
Visual Overview
UPDATE_ROLLBACK_FAILED Recovery Flow
======================================
Stack update submitted
│
▼
Resource fails during update
│
▼
CloudFormation begins automatic rollback
│
├──── Rollback succeeds ──▶ UPDATE_ROLLBACK_COMPLETE (safe)
│
└──── Rollback fails ──▶ UPDATE_ROLLBACK_FAILED (stuck!)
│
┌─────┴──────┐
│ Diagnose │
│ stack │
│ events │
└─────┬──────┘
│
┌─────┴──────┐
│ Fix cause │
│ out of │
│ band │
└─────┬──────┘
│
┌─────┴──────────────┐
│ continue-update- │
│ rollback │
│ (--resources-to- │
│ skip if needed) │
└─────┬──────────────┘
│
▼
UPDATE_ROLLBACK_COMPLETE
│
Reconcile any
skipped resources
Key Takeaway
UPDATE_ROLLBACK_FAILED happens when a CloudFormation stack update fails, CloudFormation tries to roll back, and the rollback itself cannot finish. The three causes are drift (someone modified a resource outside CloudFormation), an unreachable resource, and insufficient permissions on the deployment role. The stack is stuck until you resolve it.
Recovery is deliberate: diagnose by reading stack events, fix the underlying cause out of band, then call continue-update-rollback. If a resource cannot be reconciled, skip it with --resources-to-skip, but remember that skipping leaves CloudFormation's record out of sync, so you must reconcile afterward.
At National Bank of India this happened when an on-call engineer hand-edited a security group during an incident. The fix was to reconcile the security group, add a missing permission, and continue the rollback. The lasting fix was prevention: forbid manual console edits, run drift detection before updates, and use explicit least-privilege service roles.
Remember one sentence: diagnose from stack events, fix the cause out of band, continue the rollback, and prevent it by forbidding manual edits to CloudFormation-managed resources.
---