Skip to content

Need the LOST DATA: CloudFormation SAM deploy deleted an existing DynamoDB table from another stack

-1

Hi everyone,

I’m using AWS SAM with a template.yaml that defines several Lambda functions and a DynamoDB table. During a recent deployment, my new template.yaml did not include the DynamoDB table resource (it was previously defined in an older version of the different stack).

After deploying this updated template with sam deploy, CloudFormation deleted the DynamoDB table, even though that table was also being referenced and used in another stack.

This came as a surprise because I expected the stack update to skip unchanged or missing resources rather than delete a table shared across stacks.

My questions are:

Why did CloudFormation/SAM delete the table when it wasn’t in the updated template?

Is there a way to prevent CloudFormation from deleting existing resources when they’re omitted from the template (for example, by marking them as retained or imported)?

What’s the safest pattern for referencing shared resources like DynamoDB tables across multiple stacks?

Thanks in advance for any guidance or best practices on preventing this kind of accidental deletion!

Also, if you could help me in getting that data back - I would be grateful. It was a production table with 2 weeks of data in it. I have created the new table with same name - hoping it would bring back old data but no luck.

Please help! Note: I do not have backup or PITR on the previous table

Regards, Saujanya

1 Answer
0

When you remove a resource from a CloudFormation template and deploy the updated template, CloudFormation interprets this as an instruction to delete that resource. This is the default behavior - if a resource was previously managed by a stack and is no longer defined in the template, CloudFormation will delete it during the update.

To prevent this kind of accidental deletion in the future, you have several options:

  1. Use DeletionPolicy attribute: For resources you want to retain even when removed from a template, add the DeletionPolicy: Retain attribute to the resource definition. This tells CloudFormation to keep the resource even when it's removed from the template or when the stack is deleted.

  2. Use UpdateReplacePolicy attribute: This attribute controls what happens when a resource needs to be replaced during an update. Setting UpdateReplacePolicy: Retain ensures that the old resource is kept when replaced.

  3. Split your infrastructure: Keep stateful resources like databases in separate stacks from frequently changing components like Lambda functions. This isolation helps prevent unintended replacements or deletions.

  4. Use explicit resource names: When creating resources like DynamoDB tables, specify an explicit name rather than letting CloudFormation generate one. This makes it easier to identify and reference resources across stacks.

For sharing resources across stacks, the recommended pattern is:

  • Create the shared resources in one "foundation" stack with DeletionPolicy: Retain
  • In other stacks, import these resources using CloudFormation's cross-stack references or by directly referencing them by name

Unfortunately, regarding your lost data, if the table was deleted and you've already created a new table with the same name, the original data is likely gone. DynamoDB doesn't automatically restore data when you recreate a table with the same name. If you had enabled Point-in-Time Recovery for the table, you might be able to restore from a backup. Otherwise, check if there are any automatic backups available in the AWS Backup service that might have captured your table before deletion.

For future protection, I recommend:

  1. Always use DeletionPolicy: Retain for production databases
  2. Enable Point-in-Time Recovery for important DynamoDB tables
  3. Set up regular backups using AWS Backup
  4. Consider using the RetainExceptOnCreate DeletionPolicy, which behaves like Retain for all stack operations except for the initial creation
    Sources
    DeletionPolicy attribute - AWS CloudFormation
    UpdateReplacePolicy attribute - AWS CloudFormation
    CDK Version Upgrade Causing DynamoDB Table Replacement and Unreferenced Historical Data | AWS re:Post
    Enable Deletion Protection on DynamoDb Globaltables using Cloudformation YAML | AWS re:Post

answered 9 months ago

AWS
EXPERT

reviewed 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.