Why does the new Amazon EBS volume I created from an encrypted snapshot not exist?

Short description

When you create an Amazon EBS volume from a snapshot, two operations run:

1. First, Amazon EBS initiates the volume creation. This returns a volume ID, and sets the volume state to creating (as returned in the API/AWS CLI output). This means that the CreateVolume API is valid and registered successfully.
2. Next, an asynchronous call initiates to validate the KMS key used to encrypt and decrypt the volume you are creating.

If the KMS validation succeeds, then the volume state is set to available, and the EBS volume becomes accessible. But, if the specified KMS key ID, Alias, or ARN are not valid, then the action might appear as complete, but eventually fails with no errors returned.

Note: The KMS validation for CreateVolume is asynchronous. For more information, see create-volume and Encrypting EBS resources.

You might only notice the problem when you try to attach or access the EBS volume. You then find that it doesn't exist, despite the CreateVolume API returning a volume ID. Also, if you check the AWS CloudTrail logs, you will not see any errors.

Example of issue

This example shows the CreateVolume API in use with an invalid alias for the KMS key. The CreateVolume API succeeds and returns a volume ID, and then sets the volume state to creating. Because the alias for the KMS key isn't valid, the asynchronous authentication fails. This causes the whole operation to fail. When you check the AWS CloudTrail logs for the create volume event, no errors are found because the CreateVolume operation succeeded.

$ aws ec2 create-volume --volume-type gp2 --availability-zone eu-west-1c --encrypted --kms-key-id hana --snapshot-id snap-0a27fe340500641d9 
{
    "AvailabilityZone": "eu-west-1c", 
    "MultiAttachEnabled": false, 
    "Tags": [], 
    "Encrypted": true, 
    "VolumeType": "gp2", 
    "VolumeId": "vol-043fe27d0ccf74b36", 
    "State": "creating", 
    "KmsKeyId": "hana", 
    "SnapshotId": "snap-0a27fe340500641d9", 
    "Iops": 100, 
    "CreateTime": "2020-10-06T18:03:09.000Z", 
    "Size": 8
}

AWS CloudTrail logs:

responseElements": {
        "requestId": "8677d3cd-ad1d-4866-95f5-375d92a35813",
        "volumeId": "vol-043fe27d0ccf74b36",
        "size": "8",
        "snapshotId": "snap-0a27fe340500641d9",
        "zone": "eu-west-1c",
        "status": "creating",
        "createTime": 1602007389000,
        "volumeType": "gp2",
        "iops": 100,
        "encrypted": true,
        "masterEncryptionKeyId": "hana",
        "tagSet": {},
        "multiAttachEnabled": false
    },
    "requestID": "8677d3cd-ad1d-4866-95f5-375d92a35813",
    "eventID": "bd4216df-ba39-425e-b272-936212ae6699",
    "eventType": "AwsApiCall",
    "recipientAccountId": "864258534754"
}

When you run the describe-volume-status, you find that the volume doesn't exist:

$ aws ec2 describe-volume-status --volume-ids vol-043fe27d0ccf74b36
An error occurred (InvalidVolume.NotFound) when calling the DescribeVolumeStatus operation: 
The volume 'vol-043fe27d0ccf74b36' does not exist.

Resolution

Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.

Use the describe-volume-status API

Use the describe-volume-status API to check if the EBS volume exists.

Subscribe to the CloudWatch CreateVolume events

Subscribe to the public CloudWatch CreateVolume events for more information on volume creation failure. This is an example notification that you might receive. It notifies you of the CreateVolume CloudWatch event. From the notification, you get more insight into the event, and you can see that the CreateVolume result failed because of an Invalid keyId.

AWS Notification Message
CreateVolume <no-reply@sns.amazonaws.com>
{
"version":"0","id":"192e695f-2387-1cf0-fb1c-1cb32f047212",
"detail-type":"EBS Volume Notification","source":"aws.ec2",
"account":"12345678","time":"2020-10-06T18:03:10Z",
"region":"eu-west-1",
"resources":["arn:aws:ec2:eu-west-1:864258534754:volume/vol-043fe27d0ccf74b36"],
"detail":
{"result":"failed","cause":"Invalid keyId hana","event":"createVolume",
"request-id":"8677d3cd-ad1d-4866-95f5-375d92a35813"}
}

Note: If you create an EBS volume from an encrypted snapshot, it can also fail to create for following reasons:

• The AWS Identity and Access Management (IAM) user/role creating the volume doesn't have sufficient permissions to access the KMS key used to encrypt the snapshot.
• The KMS key used to encrypt the snapshot has been disabled, deleted or is not in the Region.

Related information

How can I optimize the performance of my Amazon EBS volumes?

Why can't I find the user name that created an EBS volume by searching CloudTrail events logs?