SSH broken as of 1st March

0

We have an automated codepipeline that builds packer images based on amazon linux AMI updates. This pipeline has worked flawlessly for several years. Yesterday however it broke with : ==> amazon-ebs: Waiting for instance (xxxxxxx) to become ready... ==> amazon-ebs: Using ssh communicator to connect: xx.xx.xx.xx ==> amazon-ebs: Waiting for SSH to become available... ==> amazon-ebs: Error waiting for SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [publickey none], no supported methods remain

We have changed nothing , something in the base AWS Li2 image has broken this we think.

  • Interestingly if I run the image created on a T2 micro instead of a T3 micro , I can ssh to it.

  • Please don't post any personal information in the question. If you would like to share personal information, create a support case instead.

2 Answers
0

Hello,

Thank you for asking your question in AWS re:Post. I will recommend to create a Support Case so we can investigate further.

Amazon Linux 2 Public AMIs will have necessary pre-requisites for running t3.micro or any Nitro system instances. You may also investigate the CloudTrail logs for CodePipeline or any EC2 API calls for the EC2 instance.

AWS
SUPPORT ENGINEER
answered 2 years ago
0

I raised it with support at the same time as posting on here. They came back yesterday to say there is a problem with the latest AMI , which breaks ssh on T3 , and for anyone interested here's the workaround:-

                                            ****************************************

Creating derived AMIs (by creating AMIs from EBS snapshots of EC2 instances) and not cleaning up cloud-init cached state may experience issues with cloud-init executing in AMIs derived from the amzn2-ami-hvm-2.0.20220218.0 Amazon Linux 2 AMI (released on 2022-02-28) .

We recommend the following workaround currently, until the issue is fixed:

Remove cloud-init's cached state using cloud-init clean.

Create an empty file at /etc/systemd/system/cloud-init-local.service.d/skip.conf before creating an EBS snapshot/AMI.

This file will allow AMI's derived from amzn2-ami-hvm-2.0.20220218.0 Amazon Linux 2 AMI to not run into this issue.

Example commands to run before snapshotting an AMI:

cloud-init clean mkdir -p /etc/systemd/system/cloud-init-local.service.d touch /etc/systemd/system/cloud-init-local.service.d/skip.conf

No other versions of the Amazon Linux 2 AMI contains this issue hence, you may also consider rolling back to previous AMI version if the above shared workaround is not feasible to you.

                                            *************************************************

We sincerely apologize for any inconvenience this may have caused, and regret that it has affected your production.

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions