Copied snapshot is being deleted after creation by DLM

Hello.

Who is executing the DeleteSnapshot event in the destination region?
You can find out who is executing it by checking the CloudTrail event history.
It may be possible that a service other than Data Lifecycle Manager is deleting the snapshots.
You can check this in the "User name" column as shown below.

This behavior appears to be related to how Amazon Data Lifecycle Manager (DLM) manages snapshots, particularly when there are issues with the snapshot creation or retention settings.

When DLM creates cross-region copies of snapshots, it should retain them according to your specified retention policy (in your case, 1 month). The fact that the snapshots are being deleted shortly after creation suggests there might be an issue with how DLM is managing these snapshots.

Here are some potential causes and solutions:

1. Retention Schedule Type: DLM supports two types of retention schedules - count-based and age-based. You've specified an age-based retention ({"Interval":1, "IntervalUnit": "MONTHS"}), which should keep the snapshots for one month. However, there might be an issue with how this is being applied to cross-region copies.

2. Source Instance Termination: If the source EC2 instance from which the original snapshot was taken has been terminated, this could affect how DLM manages the snapshots. According to DLM's behavior:

• For count-based retention schedules, if the source instance is terminated, DLM no longer manages the snapshots, but it doesn't automatically delete them.
• For age-based retention schedules (which you're using), DLM continues to delete snapshots on the defined schedule up to, but not including, the last snapshot.

3. Encryption and Key Access: Since you mentioned the snapshots are encrypted with customer-managed keys, verify that the AWSDataLifecycleManagerDefaultRole has proper permissions to use these keys in the target region.

4. Monitoring and Debugging: To better understand what's happening, you could:

• Set up an EventBridge rule to capture the DeleteSnapshot events with a pattern like:

{
"source": ["aws.ec2"],
"detail-type": ["AWS API Call via CloudTrail"],
"detail": {
"eventSource": ["ec2.amazonaws.com"],
"eventName": ["DeleteSnapshot"]
}
}

• This could help you identify which service or role is initiating the deletion.

5. Manual Testing: Try creating a cross-region copy of the snapshot manually to see if it persists, which would help determine if the issue is specific to DLM.

If the issue persists, you might need to review your DLM policy configuration or contact AWS Support for further assistance, as this behavior doesn't align with the expected functionality of DLM's retention policies.
Sources
Lifecycle Manger - Issue with Snapshot Management Post Instance Termination | AWS re:Post
Amazon DLM for snapshot automation | AWS re:Post
Is it possible to restore a deleted EC2 instance or snapshot from Lifecycle Backup | AWS re:Post