Skip to content

nested stack still stuck in UPDATE_ROLLBACK_IN_PROGRESS after being stuck in UPDATE_IN_PROGRESS and root stack update cancelled

0

Hi,

I currently have a nested stack stuck in UPDATE_ROLLBACK_IN_PROGRESS for some time - it was previously stuck in UPDATE_IN_PROGRESS and I was told that my attempts to cancel the stack update failed ... I then tried to cancel the root stack update and now both root and nested are stuck in UPDATE_ROLLBACK_IN_PROGRESS. The root cause appears to have been an eventbridge cron expression that included a ? instead of a * for the day, the Resource handler returned message: "Parameter ScheduleExpression is not valid. (Service: EventBridge, Status Code: 400)"

The related CodeDeploy (we used SAM in a CodeBuild to kick off the update to lambdas and schedules ... the CodeBuild is still running sam/etc) rollback started about 2 hours ago and according to CodeDeploy completed shortly after it started, also shortly after this time the UPDATE_IN_PROGRESS of aliases stopped and showed as UPDATE_FAILED with Resource update cancelled, but the nested stack stayed in UPDATE_IN_PROGRESS until cancelled as described in the first paragraph.

I would like to have the nested (and root) stack's status unblocked so we can re-try an update with the correct cron expression or delete the stacks if necessary... ...an outlook for AWS reducing timeouts/giving more options to work around cloudformation "deadlocks" like this would be good (searching previous questions appears to indicate multi-hour / multi-day / weeklong+ waits and few signs of improvement over the years).

Cheers

1 Answer
0

I understand your frustration with the stuck CloudFormation stacks. This is a challenging situation, but there are a few steps you can try to resolve it:

  1. First, it's important to note that the rollback process can take some time, especially for complex stacks with multiple resources. The system is trying to revert all changes made during the failed update.

  2. Since both the root and nested stacks are in UPDATE_ROLLBACK_IN_PROGRESS state, you should wait for the rollback to complete naturally if possible. CloudFormation is designed to eventually resolve these situations, though it can sometimes take hours.

  3. If the rollback doesn't complete after a reasonable amount of time (e.g., several hours), you can try using the ContinueUpdateRollback operation. This can be done through the AWS CLI with the 'continue-update-rollback' command, specifying your stack name. This operation attempts to resume the rollback from where it got stuck.

  4. If the continue-update-rollback operation doesn't work or isn't applicable (as it wasn't in your previous attempt), your next option would be to contact AWS Support. They have additional tools and permissions to help unstick CloudFormation stacks.

  5. In the worst-case scenario, if none of the above steps work, you might need to consider deleting and recreating the stacks. However, this should be a last resort and done carefully to avoid data loss.

Regarding your request for an outlook on AWS improving this situation, while I don't have specific information on AWS's plans, it's worth noting that they continually work on improving their services based on customer feedback. Issues like these are certainly on their radar, but complex distributed systems like CloudFormation can have intricate failure modes that are challenging to completely eliminate.

For future deployments, consider implementing additional safeguards:

  • Use CloudFormation change sets to preview changes before applying them.
  • Implement stricter validation on your templates, especially for things like cron expressions.
  • Consider using a staging environment to test updates before applying them to production.

These practices can help reduce the likelihood of encountering similar issues in the future.

Sources
Use ContinueUpdateRollback with a CLI - AWS SDK Code Examples
Stack status of UPDATE_ROLLBACK_COMPLETE but unable to update stack | AWS re:Post

answered 2 years ago

  • Previous issues (eg, with permissions) have taken less than 5 minutes to rollback. It looks like "failures" in Eventbridge cron expressions received less testing by AWS and are subject to such deadlocks. Re-reading https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-scheduled-rule-pattern.html#eb-cron-expressions indicated that our use of ? instead of * should have been permitted, leaving us at a loss as to the issue with our cron expression which otherwise seemed valid - and gave us little hope that another attempt would work (leading to another a multi-hour delay using the stack/trying again). Manual testing of the original cron expression added as a trigger to a Lambda also gave the "ScheduleExpression is not valid" response. Manual testing of using a * instead of ? worked ... and eventually after we noted that the aforementioned article explained somewhat confusingly in Limitations: "You can't specify the Day-of-month and Day-of-week fields in the same cron expression. If you specify a value or a * (asterisk) in one of the fields, you must use a ? (question mark) in the other." This was more clearly stated on https://stackoverflow.com/questions/39482314/parameter-scheduleexpression-is-not-valid as "don't miss this important wildcard note... You cannot use * in both the Day-of-month and Day-of-week fields. If you use it in one, you must use ? in the other." (we deduced we cannot use ? in both) however updates/rollbacks should not hang for hours due to this!

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.