Stuck updating Cloud Formation because of missing Lambda for a custom resource.

Question

Hey all. We were deploying an ssm parameter using an AWS Custom Resource in the CDK. When triggering a new deploy, it got stuck trying to delete the old SSM parameter (see logs below).
ProdCoreStack | 1/5 | 10:25:20 AM | DELETE_IN_PROGRESS   | AWS::EC2::EIPAssociation              | ProdBastionEIPAssoc Requested update requires the replacement of the existing resource; deleting existing resource, then creating a new one.
ProdCoreStack | 1/5 | 10:25:20 AM | UPDATE_FAILED        | Custom::AWS                           | PRODBastionHostSSMParameter/Resource/Default (PRODBastionHostSSMParameter49676BFC) Function not found: arn:aws:lambda:***:975050170068:function:ProdCoreStack-AWS679f53fac002430cb0da5b7982bd22872-1sJMPyCnBfuv (Service: AWSLambda; Status Code: 404; Error Code: ResourceNotFoundException; Request ID: fbf1c9f6-42bb-4f41-8d26-815194de5184; Proxy: null)

I've refactored our CDK code to deploy using the built in SSM class in the CDK, but when in UPDATE_COMPLETE_CLEANUP_IN_PROGRESS it still gets stuck trying to delete that old parameter (see logs below). Any ideas how we can force the update here? 
ProdCoreStack | 15/51 | 3:03:09 PM | UPDATE_COMPLETE_CLEA | AWS::CloudFormation::Stack            | ProdCoreStack 
ProdCoreStack | 15/51 | 3:03:10 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::CustomResource   | PRODDocumentsBucketArnSSMParameter3D6EB668 
ProdCoreStack | 15/51 | 3:03:10 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::CustomResource   | PRODDbPortSSMParameterA2B80081 
ProdCoreStack | 15/51 | 3:03:10 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::CustomResource   | ProdApiGatewayProdApiDnsAliasPRODApiRegionalHostedZoneIdSSMParameter69D4710F 
ProdCoreStack | 15/51 | 3:03:10 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::CustomResource   | PRODApiBaseUrlSSMParameter113BE6EE 
ProdCoreStack | 15/51 | 3:03:10 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::CustomResource   | ProdApiGatewayProdApiDnsAliasPRODApiRegionalDomainSSMParameter8B7D1B0B 
ProdCoreStack | 15/51 | 3:03:10 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::CustomResource   | ProdApiGatewayPRODApiHttpApiIdSSMParameterCE53381C 
ProdCoreStack | 15/51 | 3:03:10 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::CustomResource   | PRODDbHostSSMParameter379968A7 
ProdCoreStack | 15/51 | 3:03:10 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::CustomResource   | ProdApiGatewayProdApiDnsAliasPRODApiCertificateArnSSMParameterB74D3A21 
ProdCoreStack | 15/51 | 3:03:10 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::CustomResource   | ProdApiGatewayProdApiDnsAliasPRODApiCustomDomainUrlSSMParameter30A8C5F0 
ProdCoreStack | 15/51 | 3:03:10 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::CustomResource   | PRODRedisUrlSSMParameter227B44F4 
ProdCoreStack | 15/51 | 3:03:10 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::CustomResource   | PRODBastionUserSSMParameter07784FB0 
ProdCoreStack | 15/51 | 3:03:10 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::CustomResource   | ProdApiGatewayProdApiDnsAliasPRODApiIntermediateDomainSSMParameterFBB2CAE7 
ProdCoreStack | 15/51 | 3:03:10 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::CustomResource   | PRODBastionHostSSMParameter49676BFC 
[15:03:16] Stack ProdCoreStack has an ongoing operation in progress and is not stable (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)
[15:03:21] Stack ProdCoreStack has an ongoing operation in progress and is not stable (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)
[15:03:27] Stack ProdCoreStack has an ongoing operation in progress and is not stable (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)
[15:03:32] Stack ProdCoreStack has an ongoing operation in progress and is not stable (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)
[15:03:38] Stack ProdCoreStack has an ongoing operation in progress and is not stable (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)
[15:03:43] Stack ProdCoreStack has an ongoing operation in progress and is not stable (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)
[15:03:49] Stack ProdCoreStack has an ongoing operation in progress and is not stable (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)
[15:03:54] Stack ProdCoreStack has an ongoing operation in progress and is not stable (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)
[15:03:59] Stack ProdCoreStack has an ongoing operation in progress and is not stable (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)
[15:04:05] Stack ProdCoreStack has an ongoing operation in progress and is not stable (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)
[15:04:10] Stack ProdCoreStack has an ongoing operation in progress and is not stable (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)
[15:04:16] Stack ProdCoreStack has an ongoing operation in progress and is not stable (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)
[15:04:21] Stack ProdCoreStack has an ongoing operation in progress and is not stable (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)
[15:04:27] Stack ProdCoreStack has an ongoing operation in progress and is not stable (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)
... this goes on indefinitely

This is currently blocking our prod pipeline. Thanks!

re:Post Agent · Answer

Your CloudFormation stack is stuck in the UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state because it's trying to delete custom resources that were created using AWS Custom Resources in CDK, but the Lambda function that was responsible for handling these custom resources no longer exists.

When CloudFormation tries to delete a custom resource, it sends a delete request to the Lambda function specified in the ServiceToken property of that resource. In your case, the error message indicates that the Lambda function `ProdCoreStack-AWS679f53fac002430cb0da5b7982bd22872-1sJMPyCnBfuv` cannot be found (404 ResourceNotFoundException).

This situation typically happens when:
1. You've refactored your CDK code to use a different approach (as you mentioned, switching to the built-in SSM class)
2. The Lambda function that was handling the custom resources was deleted or replaced before CloudFormation could complete the cleanup process

To resolve this issue, you have a few options:

1. Recreate the missing Lambda function temporarily:
- Create a Lambda function with the exact same name and ARN that's being referenced
- Implement a simple handler that responds to CloudFormation custom resource requests with SUCCESS
- Once the stack update completes successfully, you can delete this temporary function

2. Skip the cleanup phase by manually updating the stack status in CloudFormation:
- This is more advanced and requires AWS CLI commands to modify the stack state
- You would need to mark the stack as UPDATE_COMPLETE instead of UPDATE_COMPLETE_CLEANUP_IN_PROGRESS

3. Contact AWS Support:
- If the above options don't work, AWS Support can help manually update the stack state

For the first option, the Lambda function would need to properly handle CloudFormation custom resource requests by sending a SUCCESS response with the appropriate PhysicalResourceId. This would allow CloudFormation to complete the deletion process.

In the future, when refactoring resources that use custom resources, it's best to ensure that the Lambda functions handling those resources remain available until all cleanup operations are complete.  
**Sources**  
[Update request for CloudFormation custom resources - AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/crpg-ref-requesttypes-update.html)  
[AWS::CloudFormation::CustomResource - AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-cloudformation-customresource.html)  
[aws-cdk-lib.custom_resources module · AWS CDK](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.custom_resources-readme.html)  
[AWS::Lambda::Function - AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-lambda-function.html)

Stuck updating Cloud Formation because of missing Lambda for a custom resource.

Add your answer

Relevant content