CFM stack is stuck after failure deleting Beanstalk app

Question

Hi,
I have a CloudFormation stack which includes a Beanstalk app along with an environment. I tried to remove this whole Beanstalk app within the CFM stack by simply removing the resources within the template. However, because temination protection was enabled on the nested stack that was created for the Beanstalk app, the deletion failed. The following error appeared within the Beanstalk console.

ERROR Service:AmazonCloudFormation, Message:Stack [awseb-e-xxxxxxxxxx-stack] cannot be deleted while TerminationProtection is enabled

I fixed it by disabling termination protection on the nested stack, but the problem is that the main/root/parent stack is stuck in the UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state. At no point has CFM/Beanstalk tried to delete the nested stack again since the initial failure. Essentially the stack update has been stuck ever since and has not recovered into a useful state, effectively taking our whole stack as hostage.

So basically: Beanstalk failed to delete its nested stack due to a misconfiguration, causing the root stack to be indefinitely stuck. Apparently there is no retry logic happening, since CFM/Beanstalk only attempted the operation once (I fixed the underlying problem almost immediately). It seems like Beanstalk doesn't properly communicate to CFM that the operation failed, causing the stack update to be left hanging.

How do I get out of this state? It's been so long now that I doubt CFM is able to handle it automatically. Is it safe to manually delete the nested Beanstalk stack? I just don't want to mess anything up if I do things outside of CFM's control.

Thanks!

thio123 · Accepted Answer

Okay, so this place is absolutely dead.

I ended up taking a chance and deleted the Beanstalk environment within the AWS Console. I had to do that since CloudFormation never tried to delete the environment again after the initial attempt. I am disappointed that there is no retry mechanism when the solution is so obvious, but I guess that's just one of the many quirks of CloudFormation. Yes, it was my own mistake that caused me to end up in that situation, but I hoped and expected CFM to handle it a bit more gracefully instead of just getting stuck indefinitely. Disappointing!

So delete the environment manually and CFM will "understand" this and proceed with the stack update afterwards. At least that's what happened for me.

CFM stack is stuck after failure deleting Beanstalk app

Añade tu respuesta

Contenido relevante