CFM stack is stuck after failure deleting Beanstalk app
Hi, I have a CloudFormation stack which includes a Beanstalk app along with an environment. I tried to remove this whole Beanstalk app within the CFM stack by simply removing the resources within the template. However, because temination protection was enabled on the nested stack that was created for the Beanstalk app, the deletion failed. The following error appeared within the Beanstalk console.
ERROR Service:AmazonCloudFormation, Message:Stack [awseb-e-xxxxxxxxxx-stack] cannot be deleted while TerminationProtection is enabled
I fixed it by disabling termination protection on the nested stack, but the problem is that the main/root/parent stack is stuck in the UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state. At no point has CFM/Beanstalk tried to delete the nested stack again since the initial failure. Essentially the stack update has been stuck ever since and has not recovered into a useful state, effectively taking our whole stack as hostage.
So basically: Beanstalk failed to delete its nested stack due to a misconfiguration, causing the root stack to be indefinitely stuck. Apparently there is no retry logic happening, since CFM/Beanstalk only attempted the operation once (I fixed the underlying problem almost immediately). It seems like Beanstalk doesn't properly communicate to CFM that the operation failed, causing the stack update to be left hanging.
How do I get out of this state? It's been so long now that I doubt CFM is able to handle it automatically. Is it safe to manually delete the nested Beanstalk stack? I just don't want to mess anything up if I do things outside of CFM's control.
Okay, so this place is absolutely dead.
I ended up taking a chance and deleted the Beanstalk environment within the AWS Console. I had to do that since CloudFormation never tried to delete the environment again after the initial attempt. I am disappointed that there is no retry mechanism when the solution is so obvious, but I guess that's just one of the many quirks of CloudFormation. Yes, it was my own mistake that caused me to end up in that situation, but I hoped and expected CFM to handle it a bit more gracefully instead of just getting stuck indefinitely. Disappointing!
So delete the environment manually and CFM will "understand" this and proceed with the stack update afterwards. At least that's what happened for me.
Can't run 'rails console' on Elastic Beanstalk Amazon Linux 2Accepted Answerasked 3 months ago
CFM stack is stuck after failure deleting Beanstalk appAccepted Answerasked 3 months ago
Elastic Beanstalk Environment is stuck on Terminating for 2 hours nowasked 2 years ago
How to deploy to single instance Elastic Beanstalk worker environment using CodePipeline?asked 6 months ago
Restart Elastic Beanstalk App Server when Health is Severeasked 4 months ago
Elastic Beanstalk stuck aborting current environment operationasked 4 months ago
Can't delete EBS app or envasked 16 days ago
How to pass the Amplify app ID to a function? How to do app introspection from backend functions?Accepted Answerasked 2 months ago
Cloudformation Stack Stuck in UPDATE_IN_PROGRESSasked 2 years ago
Beanstalk blazor server app “Error: Cannot send data if the connection...asked a year ago