Launch Template Incorrectly Creating BlockDeviceMappings


I have a weird problem that cropped up in the last couple of weeks.

Problem I have an older launch template which I will call "old_lt". We've used it for about 2 years. Before, when I made a version update to old_lt, I could see a volume listed under "Storage" for the snapshot associated with the LT AMI. If I inspect this LT using "aws ec2 describe-launch-template-versions...", I can see that there is an BlockDeviceMapping for the snapshot and 4 BlockingDeviceMappings for NoDevice entries. These NoDevice entries have been present for the entire two years and have never caused a problem. Now, when I update the LT, there is no storage entry for the snapshot volume. If I attempt to launch an instance from the LT, an EC2 instance is launched with no volume attached to it and get's stuck pending. If I create a new LT using old_lt as the source template, the same issue occurs. If I use the CLI to inspect the template, I can see the four NoDevice entries for BlockDeviceMappings, but no entry for the snapshot volume.

Solution The only solution I have found is to completely rebuild the Launch Template from scratch, which I will call "new_lt". When I completely rebuild the LT without using a source template, I still do not see an entry for the snapshot volume under "Storage", but I see the same four NoDevice entries. When I inspect new_lt using "aws ec2 describe-launch-template-versions..." I do not see any entries for BlockDeviceMappings.

Question I really don't want to rebuild every launch template that we have from scratch. This is not only a time suck and error prone, but this is going to make a mess since we need our existing LTs for rollback. Does anyone have any idea what happened in the last month or so to cause this? This has never been a problem before. The behavior makes me suspect that this is correlated to a recent changes in the LT Management Console interface, but I have been unable to pin down the exact cause.

Additional Information When I modify templates through the Management Console now, I can see "AMI Volumes are not included in the template unless modified" under "EBS Volumes". Is this new? I can't say for sure, but I don't feel like this has been there the entire time I've been on my project.

asked 4 months ago213 views
1 Answer
Accepted Answer

Answering my own question, which took me quite a lot of time to figure out. The issue is that when I am updating the LT through the Management Console, if I don't update the volume, then the LT does not store a BlockDeviceMapping for that volume. This appears to be a new behavior. The problem was that the LT does store BlockDeviceMappings for the pre-existing NoDevice entries. Since BlockDeviceMappings are stored with the LT, those mappings are used by the LT, not the default mappings. Since there is no longer a mapping defined for the root volume, an instance is launched without one. This is obviously a problem.

To resolve this, I created a new version of the template using the CLI and JQ which deletes the BlockDeviceMappings.

aws ec2 create-launch-template-version --launch-template-id <lt_id> --launch-template-data "$(aws ec2 describe-launch-template-versions --versions <latest_version> --launch-template-id <lt_id> --query "LaunchTemplateVersions[0].LaunchTemplateData" | jq 'del(.BlockDeviceMappings)')"

Hope this helps someone out there who is as befuddled as I was.

answered 4 months ago
profile picture
reviewed 17 days ago
profile picture
reviewed 4 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions