How do I delete a Lambda-backed custom resource that's stuck in DELETE_FAILED status or DELETE_IN_PROGRESS status in CloudFormation?

4 minute read

My AWS Lambda-backed custom resource is stuck in DELETE_FAILED status or DELETE_IN_PROGRESS status in AWS CloudFormation. I want to delete my custom resource.

Short description

Choose one of the following solutions:

Delete a custom resource that's stuck in DELETE_FAILED status
Delete a custom resource that's stuck in DELETE_IN_PROGRESS status

Resolution

Delete a custom resource that's stuck in DELETE_FAILED status

You get an error message if you try to delete a stack when your Lambda-backed custom resource is missing the logic to handle delete requests. You can also get an error if your custom resource contains incorrect delete logic. The error message is: "Custom Resource failed to stabilize in expected time." The status of the stack changes to DELETE_FAILED.

To delete your stack, complete the following steps:

1. Open the CloudFormation console.

2. Choose the stack that contains your custom resource that's stuck in DELETE_FAILED status.

3. Choose Actions, and then choose Delete Stack.

4. In the pop-up window that provides a list of resources to retain, choose the custom resource that's stuck in DELETE_FAILED status. Then, choose Delete.

5. Choose Actions, and then choose Delete Stack.

The status of your stack changes to DELETE_COMPLETE.

Note: Your custom resource isn't a physical resource, so you don't have to clean up your custom resource after stack deletion.

Delete a custom resource that's stuck in DELETE_IN_PROGRESS status

If your stack gets stuck in DELETE_IN_PROGRESS status while deleting a custom resource, then your stack could be missing the logic to handle delete requests. Or, your stack could include incorrect delete logic.

To force the stack to delete, you must manually send a SUCCESS signal. The signal requires the ResponseURL and RequestId values, which are both included in the event that's sent from CloudFormation to Lambda.

1. Confirm that your Lambda function logs the event details.

If your Lambda function doesn't include the logic to print the event to logs, then you can't get the ResponseURL and RequestId values. These values are required for sending a manual signal. In this case, you must wait for approximately one hour for the CloudFormation stack to timeout and go to the DELETE_FAILED state. Then, complete the steps in the Delete a custom resource that's stuck in DELETE_FAILED status section to delete your stack.

Note: If you already have the following logic in your Lambda function, then go to step 2.

Example logic:

exports.handler = function(event, context) {
    console.log("REQUEST RECEIVED:\n" + JSON.stringify(event));
...
}

2. In your CloudFormation template, identify the name of the Lambda function where your custom resource is sending requests. You can find the function name from the ServiceToken property of the AWS::CloudFormation::CustomResource or Custom::String resource. For example:

MyCustomResource: 
  Type: "Custom::PingTester"
  Properties: 
    ServiceToken:
      !Sub |
        arn:aws:lambda:us-east-1:111122223333:function:awsexamplelambdafunction

3. Open the Lambda console.

4. In the navigation pane, choose Functions, and then choose the function that you identified in step 1. For example, awsexamplelambdafunction is the function name in the preceding code example.

5. Choose the Monitoring tab, and then choose View logs in CloudWatch.

6. In the Amazon CloudWatch console, choose the latest log.

Note: You can see the CloudWatch logs only if your function has access to Amazon CloudWatch Logs for log streaming.

7. In the latest log, identify the event where RequestType is set to Delete, and then copy the values for RequestId, ResponseURL, StackId, LogicalResourceId, and PhysicalResourceId. For example:

Received event: {
  "RequestType": "Delete",
  "ServiceToken": "arn:aws:lambda:us-east-1:111122223333:function:awsexamplelambdafunction",
  "ResponseURL": "https://cloudformation-custom-resource-response-useast1.s3.us-east-1.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-1%3A111122223333%3Astack/awsexamplecloudformation/33ad60e0-5f25-11e9-a734-0aa6b80efab2%7CMyCustomResource%7Ce2fc8f5c-0391-4a65-a645-7c695646739?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20170313T0212304Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=QWERTYUIOLASDFGBHNZCV%2F20190415%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=dgvg36bh23mk44nj454bjb54689bg43r8v011uerehiubrjrug5689ghg94hb",
  "StackId": "arn:aws:cloudformation:us-east-1:111122223333:stack/awsexamplecloudformation/33ad60e0-5f25-11e9-a734-0aa6b80efab2",
  "RequestId": "e2fc8f5c-0391-4a65-a645-7c695646739",
  "LogicalResourceId": "MyCustomResource",
  "PhysicalResourceId": "test-MyCustomResource-1URTEVUHSKSKDFF",
  "ResourceType": "Custom::PingTester"

8. To send a SUCCESS response signal in the response object to the delete request, run the following command in your local command-line interface. Be sure to include the values that you copied from step 7.

$ curl -H 'Content-Type: ''' -X PUT -d '{
    "Status": "SUCCESS",
    "PhysicalResourceId": "test-CloudWatchtrigger-1URTEVUHSKSKDFF",
    "StackId": "arn:aws:cloudformation:us-east-1:111122223333:stack/awsexamplecloudformation/33ad60e0-5f25-11e9-a734-0aa6b80efab2
  ",
    "RequestId": "e2fc8f5c-0391-4a65-a645-7c695646739",
    "LogicalResourceId": "CloudWatchtrigger"
  }' 'https://cloudformation-custom-resource-response-useast1.s3.us-east-1.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-1%3A111122223333%3Astack/awsexamplecloudformation/33ad60e0-5f25-11e9-a734-0aa6b80efab2%7CMyCustomResource%7Ce2fc8f5c-0391-4a65-a645-7c695646739?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20170313T0212304Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=QWERTYUIOLASDFGBHNZCV%2F20190415%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=dgvg36bh23mk44nj454bjb54689bg43r8v011uerehiubrjrug5689ghg94hb
  '

The status of your custom resources changes to DELETE_COMPLETE in the CloudFormation stack events.