Bug: Custom Resource deleted due to stack rollback is not given correct PhysicalResourceId

Description: I noticed a bug on CloudFormation Custom Resources: when a custom resource creation is cancelled by CloudFormation because another update fails, the custom resource handler receives the wrong PhysicalResourceId in the Delete event during the ROLLBACK phase.

Impact: This is a problem because, in line with AWS recommendations, the PhysicalResourceId is often the ID of a resource in another system, and if the Delete handler is not passed the correct ID, then it cannot delete the resource that has already been created.

Reproduction: To illustrate it I created a simple lambda resource handler that delays for a number of seconds (passed as Delay property) and then succeeds or fails (based on the Succeed property). With the result it also sends a custom PhysicalResourceId.

I created a template with three custom resources:

Succeeding1: waits 1 second and returns a success response to CloudFormation
Failing: waits 5 seconds and returns a failure response to CloudFormation
Succeeding2: waits 10 seconds and returns a success response to CloudFormation

This gives the following events:

Failing                         CREATE_IN_PROGRESS
Succeeding1                     CREATE_IN_PROGRESS
Succeeding2                     CREATE_IN_PROGRESS
Succeeding1                     CREATE_IN_PROGRESS
    Resource creation Initiated
Succeeding1                     CREATE_COMPLETE
Failing                         CREATE_IN_PROGRESS
    Resource creation Initiated
Failing                         CREATE_FAILED
    Received response status [FAILED] from custom resource.
Succeeding2                     CREATE_FAILED
    Resource creation cancelled
TestStack                       ROLLBACK_IN_PROGRESS
    The following resource(s) failed to create: [Failing, Succeeding2]. Rollback requested by user.
Succeeding2                     DELETE_IN_PROGRESS
Failing                         DELETE_IN_PROGRESS
Succeeding1                     DELETE_IN_PROGRESS
Failing                         DELETE_COMPLETE
Succeeding1                     DELETE_COMPLETE
Succeeding2                     DELETE_COMPLETE
TestStack                       ROLLBACK_COMPLETE

Note that Succeeding2 fails to create with "Resource creation cancelled" as the reason. Actually, according to the logs for the resource handler, Succeeding2 is successfully created, but when it is deleted it is not sent the PhysicalResourceId returned by the Create handler.

Below are the edited logs from the resource handler. Note that when the resource with logical ID Success2 is deleted, the handler is passed a PhysicalResourceId of TestStack-Succeeding2-18K7KJ8WCITKL, even though the Create handler sent a value of success2 to CloudFormation.

2022-11-19T19:25:34.842Z	5c0c23c4-babd-47ea-9a9d-730b669ec1d0	INFO	REQUEST Create {
  RequestType: 'Create',
  ServiceToken: 'arn:aws:lambda:eu-central-1:****:function:TestStack-TestResourceHandler-4uzRRSEzppyu',
  ResponseURL: '****',
  StackId: 'arn:aws:cloudformation:eu-central-1:****:stack/TestStack/d82bc950-683f-11ed-9330-021d95dc4328',
  RequestId: '79864543-61a7-4956-b53b-b97b4d624612',
  LogicalResourceId: 'Succeeding1',
  ResourceType: 'Custom::TestResource',
  ResourceProperties: {
    ServiceToken: 'arn:aws:lambda:eu-central-1:****:function:TestStack-TestResourceHandler-4uzRRSEzppyu',
    Id: 'succeeding1',
    Delay: '1',
    Succeed: 'true'
  }
}

2022-11-19T19:25:34.932Z	b81e3ec0-a42f-4833-9ac0-d835e1189c64	INFO	REQUEST Create {
  RequestType: 'Create',
  ServiceToken: 'arn:aws:lambda:eu-central-1:****:function:TestStack-TestResourceHandler-4uzRRSEzppyu',
  ResponseURL: '****',
  StackId: 'arn:aws:cloudformation:eu-central-1:****:stack/TestStack/d82bc950-683f-11ed-9330-021d95dc4328',
  RequestId: '2c124be0-10c5-4df1-a34f-d1ba36244b8a',
  LogicalResourceId: 'Failing',
  ResourceType: 'Custom::TestResource',
  ResourceProperties: {
    ServiceToken: 'arn:aws:lambda:eu-central-1:****:function:TestStack-TestResourceHandler-4uzRRSEzppyu',
    Id: 'failing',
    Delay: '5',
    Succeed: 'false'
  }
}

2022-11-19T19:25:34.955Z	8b7821bc-731b-4b21-b06e-eb18f93184f2	INFO	REQUEST Create {
  RequestType: 'Create',
  ServiceToken: 'arn:aws:lambda:eu-central-1:****:function:TestStack-TestResourceHandler-4uzRRSEzppyu',
  ResponseURL: '****',
  StackId: 'arn:aws:cloudformation:eu-central-1:****:stack/TestStack/d82bc950-683f-11ed-9330-021d95dc4328',
  RequestId: '138ce7f4-9814-49e4-aa5b-b3723069c91a',
  LogicalResourceId: 'Succeeding2',
  ResourceType: 'Custom::TestResource',
  ResourceProperties: {
    ServiceToken: 'arn:aws:lambda:eu-central-1:****:function:TestStack-TestResourceHandler-4uzRRSEzppyu',
    Id: 'succeeding2',
    Delay: '10',
    Succeed: 'true'
  }
}

2022-11-19T19:25:35.844Z	5c0c23c4-babd-47ea-9a9d-730b669ec1d0	INFO	RESPONSE {
  Status: 'SUCCESS',
  LogicalResourceId: 'Succeeding1',
  PhysicalResourceId: 'succeeding1',
  Reason: 'Success: Create',
  RequestId: '79864543-61a7-4956-b53b-b97b4d624612',
  StackId: 'arn:aws:cloudformation:eu-central-1:****:stack/TestStack/d82bc950-683f-11ed-9330-021d95dc4328'
}

2022-11-19T19:25:39.963Z	b81e3ec0-a42f-4833-9ac0-d835e1189c64	INFO	RESPONSE {
  Status: 'FAILED',
  LogicalResourceId: 'Failing',
  PhysicalResourceId: 'failing',
  Reason: 'Failure: Create',
  RequestId: '2c124be0-10c5-4df1-a34f-d1ba36244b8a',
  StackId: 'arn:aws:cloudformation:eu-central-1:****:stack/TestStack/d82bc950-683f-11ed-9330-021d95dc4328'
}

2022-11-19T19:25:44.958Z	8b7821bc-731b-4b21-b06e-eb18f93184f2	INFO	RESPONSE {
  Status: 'SUCCESS',
  LogicalResourceId: 'Succeeding2',
  PhysicalResourceId: 'succeeding2',
  Reason: 'Success: Create',
  RequestId: '138ce7f4-9814-49e4-aa5b-b3723069c91a',
  StackId: 'arn:aws:cloudformation:eu-central-1:****:stack/TestStack/d82bc950-683f-11ed-9330-021d95dc4328'
}

2022-11-19T19:25:58.222Z	ed3edc75-d612-4795-8a84-37dd9dbab15a	INFO	REQUEST Delete {
  RequestType: 'Delete',
  ServiceToken: 'arn:aws:lambda:eu-central-1:****:function:TestStack-TestResourceHandler-4uzRRSEzppyu',
  ResponseURL: '****',
  StackId: 'arn:aws:cloudformation:eu-central-1:****:stack/TestStack/d82bc950-683f-11ed-9330-021d95dc4328',
  RequestId: 'f17b0b04-c777-4fc8-8fcf-c6bafc19fe24',
  LogicalResourceId: 'Failing',
  PhysicalResourceId: 'failing',
  ResourceType: 'Custom::TestResource',
  ResourceProperties: {
    ServiceToken: 'arn:aws:lambda:eu-central-1:****:function:TestStack-TestResourceHandler-4uzRRSEzppyu',
    Id: 'failing',
    Delay: '5',
    Succeed: 'false'
  }
}

2022-11-19T19:25:58.222Z	ed3edc75-d612-4795-8a84-37dd9dbab15a	INFO	RESPONSE {
  Status: 'SUCCESS',
  LogicalResourceId: 'Failing',
  PhysicalResourceId: 'failing',
  Reason: 'Success: Delete',
  RequestId: 'f17b0b04-c777-4fc8-8fcf-c6bafc19fe24',
  StackId: 'arn:aws:cloudformation:eu-central-1:****:stack/TestStack/d82bc950-683f-11ed-9330-021d95dc4328'
}

2022-11-19T19:25:58.338Z	504e4fd4-86c5-4f31-8be1-3da66cfc43e9	INFO	REQUEST Delete {
  RequestType: 'Delete',
  ServiceToken: 'arn:aws:lambda:eu-central-1:****:function:TestStack-TestResourceHandler-4uzRRSEzppyu',
  ResponseURL: '****',
  StackId: 'arn:aws:cloudformation:eu-central-1:****:stack/TestStack/d82bc950-683f-11ed-9330-021d95dc4328',
  RequestId: 'c0b51a3c-278c-4999-8ac7-f48bea2c9378',
  LogicalResourceId: 'Succeeding1',
  PhysicalResourceId: 'succeeding1',
  ResourceType: 'Custom::TestResource',
  ResourceProperties: {
    ServiceToken: 'arn:aws:lambda:eu-central-1:****:function:TestStack-TestResourceHandler-4uzRRSEzppyu',
    Id: 'succeeding1',
    Delay: '1',
    Succeed: 'true'
  }
}

2022-11-19T19:25:58.338Z	504e4fd4-86c5-4f31-8be1-3da66cfc43e9	INFO	RESPONSE {
  Status: 'SUCCESS',
  LogicalResourceId: 'Succeeding1',
  PhysicalResourceId: 'succeeding1',
  Reason: 'Success: Delete',
  RequestId: 'c0b51a3c-278c-4999-8ac7-f48bea2c9378',
  StackId: 'arn:aws:cloudformation:eu-central-1:****:stack/TestStack/d82bc950-683f-11ed-9330-021d95dc4328'
}

2022-11-19T19:25:58.517Z	6cd52109-1fb6-403c-9289-6da295829d3c	INFO	REQUEST Delete {
  RequestType: 'Delete',
  ServiceToken: 'arn:aws:lambda:eu-central-1:****:function:TestStack-TestResourceHandler-4uzRRSEzppyu',
  ResponseURL: '****',
  StackId: 'arn:aws:cloudformation:eu-central-1:****:stack/TestStack/d82bc950-683f-11ed-9330-021d95dc4328',
  RequestId: '115674d4-53c4-4845-a910-06afb547fd11',
  LogicalResourceId: 'Succeeding2',
  PhysicalResourceId: 'TestStack-Succeeding2-18K7KJ8WCITKL',
  ResourceType: 'Custom::TestResource',
  ResourceProperties: {
    ServiceToken: 'arn:aws:lambda:eu-central-1:****:function:TestStack-TestResourceHandler-4uzRRSEzppyu',
    Id: 'succeeding2',
    Delay: '10',
    Succeed: 'true'
  }
}

2022-11-19T19:25:58.517Z	6cd52109-1fb6-403c-9289-6da295829d3c	INFO	RESPONSE {
  Status: 'SUCCESS',
  LogicalResourceId: 'Succeeding2',
  PhysicalResourceId: 'succeeding2',
  Reason: 'Success: Delete',
  RequestId: '115674d4-53c4-4845-a910-06afb547fd11',
  StackId: 'arn:aws:cloudformation:eu-central-1:****:stack/TestStack/d82bc950-683f-11ed-9330-021d95dc4328'
}

Topics

Management & Governance

Relevant content

Is there a way to conditionally trigger the delete API on custom resources formed using the cloudformation template?
rePost-User-2478398
asked 2 years ago
Custom Resources Lifecycle
m66rus
asked 3 years ago
Delete a resource from CloudFormation stack
Krzysztof Slowinski
asked a year ago
Why does my stack deletion fail because of an error that occurs when deleting a custom resource?
Accepted Answer
AWS-User-9246258
asked 3 years ago
How do I resolve the AWS CloudFormation error "Cannot update a stack when a custom-named resource requires replacing"?
AWS OFFICIALUpdated a year ago
How do I troubleshoot custom resource failures in AWS CloudFormation?
AWS OFFICIALUpdated 2 months ago
What are some best practices for implementing Lambda-backed custom resources with CloudFormation?
AWS OFFICIALUpdated 2 years ago
How do I delete a Lambda-backed custom resource that's stuck in DELETE_FAILED status or DELETE_IN_PROGRESS status in CloudFormation?
AWS OFFICIALUpdated 3 years ago
What's new: AWS Resource Access Manager supports fine-grained customer managed permissions
EXPERT
Fabian
published a year ago
Creating a DynamoDB Table with CloudFormation and Adding Items at Creation Time
EXPERT
Leeroy Hannigan
published 10 months ago