How do I delete a Lambda-backed custom resource that's stuck in DELETE_FAILED status or DELETE_IN_PROGRESS status in CloudFormation?

4 minute read
1

My AWS Lambda-backed custom resource is stuck in DELETE_FAILED status or DELETE_IN_PROGRESS status in AWS CloudFormation. I want to delete my custom resource.

Resolution

Choose one of the following solutions based on the status of your resource.

Delete a custom resource that's stuck in DELETE_FAILED status

You get an error message if you try to delete a stack with a Lambda-backed custom resource that doesn't have the logic to handle delete requests. You can also get the following error message if your custom resource contains incorrect delete logic: "CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [652961ca-f24e-42e0-8f74-3f4846ba282d]. If you use the Python cfn-response module, you might need to update your Lambda function code so that CloudFormation can attach the updated version."

The stack's status changes to DELETE_FAILED, and then you get the following error message: "Custom Resource failed to stabilize in expected time."

To delete your stack, complete the following steps:

  1. Open the CloudFormation console.
  2. Choose the stack that contains your custom resource that's stuck in DELETE_FAILED status.
  3. Choose Actions, and then choose Delete Stack.
  4. In the pop-up window that provides a list of resources to retain, choose the custom resource that's stuck in DELETE_FAILED status. Then, choose Delete.
  5. Choose Actions, and then choose Delete Stack.

The status of your stack changes to DELETE_COMPLETE.

Note: Your custom resource isn't a physical resource, so you don't have to clean up your custom resource after stack deletion.

Delete a custom resource that's stuck in DELETE_IN_PROGRESS status

When your stack deletes a custom resource and it gets stuck in DELETE_IN_PROGRESS status, then your stack might not have the logic to handle delete requests. Or, your stack could include incorrect delete logic.

To force the stack to delete, you must manually send a SUCCESS signal. The signal requires the ResponseURL and RequestId values that are included with the event that's sent from CloudFormation to Lambda.

  1. In your CloudFormation template, identify the name of the Lambda function where your custom resource is sending requests. You can find the function name from the ServiceToken property of the AWS::CloudFormation::CustomResource or Custom::String resource.

    For example:

    MyCustomResource: 
      Type: "Custom::PingTester"
      Properties: 
        ServiceToken:
          !Sub |
            arn:aws:lambda:us-east-1:111122223333:function:awsexamplelambdafunction
  2. Open the Lambda console.

  3. In the navigation pane, choose Functions, and then choose the function that you identified in step 1. For example, awsexamplelambdafunction is the function name in the preceding code example.

  4. Verify that your Lambda function logs the event details to the Amazon CloudWatch log group called /aws/lambda/function_name.
    Example logic:

    exports.handler = function(event, context) {    console.log("REQUEST RECEIVED:\n" + JSON.stringify(event));
    ...
    }

    Note: If your Lambda function doesn't include the logic to print the event to CloudWatch logs, then you can't get the ResponseURL and RequestId values. You must have these values to send a manual signal. Wait for approximately one hour for the CloudFormation stack to time out and go to the DELETE_FAILED state. Then, complete the steps in the go to the Delete a custom resource that's stuck in DELETE_FAILED status section to delete your stack.

  5. Choose the Monitoring tab, and then choose View logs in CloudWatch.

  6. In the Amazon CloudWatch console, choose the latest log.

    Note: You can see the CloudWatch logs only if your function has access to Amazon CloudWatch Logs for log streaming.

  7. In the latest log, identify the event where RequestType is set to Delete, and then copy the values for RequestId, ResponseURL, StackId, LogicalResourceId, and PhysicalResourceId. For example:

    Received event: {
      "RequestType": "Delete",
      "ServiceToken": "arn:aws:lambda:us-east-1:111122223333:function:awsexamplelambdafunction",
      "ResponseURL": "https://cloudformation-custom-resource-response-useast1.s3.us-east-1.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-1%3A111122223333%3Astack/awsexamplecloudformation/33ad60e0-5f25-11e9-a734-0aa6b80efab2%7CMyCustomResource%7Ce2fc8f5c-0391-4a65-a645-7c695646739?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20170313T0212304Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=QWERTYUIOLASDFGBHNZCV%2F20190415%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=dgvg36bh23mk44nj454bjb54689bg43r8v011uerehiubrjrug5689ghg94hb",
      "StackId": "arn:aws:cloudformation:us-east-1:111122223333:stack/awsexamplecloudformation/33ad60e0-5f25-11e9-a734-0aa6b80efab2",
      "RequestId": "e2fc8f5c-0391-4a65-a645-7c695646739",
      "LogicalResourceId": "MyCustomResource",
      "PhysicalResourceId": "test-MyCustomResource-1URTEVUHSKSKDFF",
      "ResourceType": "Custom::PingTester"
  8. To send a SUCCESS response signal in the response object to the delete request, run the following command in your local command-line interface. Be sure to include the values that you copied from step 7.

    curl -H "Content-Type: ''" -X PUT -d "{\"Status\": \"SUCCESS\",\"PhysicalResourceId\": \"test-CloudWatchtrigger-1URTEVUHSKSKDFF\",\"StackId\": \"arn:aws:cloudformation:us-west-2:111122223333:stack/CRStack/5105e4b0-4a29-11ef-8c42-067744df1edb\",\"RequestId\": \"dca731ad-18ad-496c-8c77-459d904fd36b\",\"LogicalResourceId\": \"MyCustomResource\"}" "https://cloudformation-custom-resource-response-uswest2.s3-us-west-2.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-west-2%3A111122223333%3Astack/CRStack/5105e4b0-4a29-11ef-8c42-067744df1edb%7CCustomResource%7Cdca731ad-18ad-496c-8c77-459d904fd36b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20240725T020141Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIA54RCMT6SJPHCTM4H%2F20240725%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=3b1567adb5ec9415526251a26bac3a6c861ac77aa090a3ada0a4a4d109a7273d"

The status of your custom resources changes to DELETE_COMPLETE in the CloudFormation stack events.

Related information

Create custom provisioning logic with custom resources

AWS::CloudFormation::CustomResource

What are some best practices for implementing AWS Lambda-backed custom resources with AWS CloudFormation?

AWS OFFICIAL
AWS OFFICIALUpdated 5 months ago
1 Comment

If your Lambda function doesn't include the logic to print the event to logs, then you can't get the ResponseURL and RequestId values. These values are required for sending a manual signal. In this case, you must wait for approximately one hour for the CloudFormation stack to timeout and go to the DELETE_FAILED state. Then, complete the steps in the Delete a custom resource that's stuck in DELETE_FAILED status section to delete your stack.

Make sure to lower the timeout next time you launch a stack, using this property https://aws.amazon.com/about-aws/whats-new/2024/06/aws-cloudformation-dev-test-cycle-timeouts-custom-resources/. The default timeout is insane, I would like to look at the person who decided that 1 hour timeout is a good idea.

replied 6 months ago