Why is my CloudFormation stack stuck in the UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS or UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state?

4 minute read
1

My AWS CloudFormation stack is stuck in the UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS or UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state. I want to get my stack into the UPDATE_ROLLBACK_COMPLETE or UPDATE_COMPLETE state.

Short description

A CloudFormation stack gets stuck in the UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS or UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state for the following reasons:

  • CloudFormation is still removing old resources or can't remove the resources because of a dependency issue.
  • CloudFormation paused the resource cleanup because a sibling stack of a nested stack fails to update or roll back.

Resolution

Check the status of your stack

Complete the following steps:

  1. Open the CloudFormation console.
  2. In the navigation pane, choose Stacks.
  3. Choose the stack that's stuck.
  4. Check if there's a NESTED label next to the name of your stack.
  5. Under the Events tab for your stack, check if there are resources in the DELETE_IN_PROGRESS state. If the status doesn't change to DELETE_COMPLETE, then you must check the resource.

Nested stack resources are in the DELETE_IN_PROGRESS state

The resources in your nested stack can be stuck in the DELETE_IN_PROGRESS state for the following reasons:

  • Some resources, such as AWS::RDS::DBInstance and AWS::CloudFront::Distribution, require more time for CloudFormation to delete.
  • Other resources are dependent on your resource. For example, resources outside your CloudFormation stack might use your Amazon Elastic Compute Cloud (Amazon EC2) security group. Check that there are no dependencies manually created for the resource.

Note: CloudFormation tries to delete a resource three times before CloudFormation skips the resource and continues the cleanup process.

Nested stack resources aren't in the DELETE_IN_PROGRESS state

If your nested stack has no resources in the DELETE_IN_PROGRESS state, then check your root stack.

Complete the following steps:

  1. Open the CloudFormation console.
  2. In the navigation pane, choose Stacks, and then select your stack.
  3. In the Overview tab for your stack, select the stack in the Root stack section.
  4. In the Status section, check the status of the root stack.

If the root stack is in the UPDATE_IN_PROGRESS or UPDATE_ROLLBACK_IN_PROGRESS state, then CloudFormation is updating other resources that are dependent on your stack. After CloudFormation updates all the dependent resources, including resources in nested stacks, CloudFormation starts the cleanup process.

If the root stack is in the UPDATE_ROLLBACK_FAILED state, then identify the first resource that failed to roll back. Check the Events tab of the root stack on the CloudFormation console. If the resource that failed to roll back isn't a nested stack, then see How can I get my CloudFormation stack to update if it's stuck in the UPDATE_ROLLBACK_FAILED state?

The resource that failed to roll back is a nested stack

If the resource that failed to roll back is a nested stack, then complete the following steps:

  1. Identify the first resource that failed to roll back after the nested stack entered the UPDATE_ROLLBACK_IN_PROGRESS state. Check the Events tab of the nested stack on the CloudFormation console. If the resource is a nested stack, then repeat this process until you identify a non-nested stack resource that failed to roll back.
  2. Continue the rollback process to bring the stack to a stable state.

Important: If you skip a resource during a ContinueUpdateRollback operation, then CloudFormation sets the status of the specified resources to UPDATE_COMPLETE. Then, CloudFormation continues to roll back the stack. After the rollback completes, the state of the skipped resources is inconsistent with the state of the resources in the stack template. Before you update the stack again, update the resources to be consistent with each other. Otherwise, subsequent stack updates can fail and the stack can become unrecoverable.

Non-nested stack resources are in the DELETE_IN_PROGRESS state

If resources in your stack are in the DELETE_IN_PROGRESS state, then use the CloudFormation console to check the resource status. Or, check the AWS CloudTrail logs for the resource. If the resources don't change to DELETE_COMPLETE, then use the AWS Health Dashboard to look for operational issues.

Related information

Why is my CloudFormation stack stuck in an IN_PROGRESS state?

Embed stacks within other stacks using nested stacks

Update rollback failed

4 Comments

So you get down to the bottom of the troubleshooter, "If there are no resources in the DELETE_IN_PROGRESS state, then look for operational issues using the AWS Service Health Dashboard for CloudFormation. " - there's nothing reported there, but my problem is that the stack is still in UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS! Pretty useless troubleshooter!

replied 2 years ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
EXPERT
replied 2 years ago

Hi, I have a Cloudformation stack stuck in the UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state, however, all the resources are in the CREATE_COMPLETE or UPDATE_COMPLETE status. Thanks

replied a year ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied a year ago