We use Ubuntu 20.04 (ami-0c8858c090152d291) as the basis for a production ecommerce stack, and I need to move users around as part of a handover. In order to do this I am trying to ssh in to the instance using the original ami-configured instance user and AWS generated key, so I can move the user I normally log in as. This fails with the subject error in /var/log/auth.log. I have reconfirmed keys and user many times obviously.
This appears to be related to AuthorizedKeysCommand fails on Ubuntu 20.04, which blames the package ec2-instance-connect. We keep instances up to date, so I suspect this package was installed as part of a post-install security update. The above-linked GitHub thread suggests:
# rm /usr/lib/systemd/system/ssh.service.d/ec2-instance-connect.conf
# systemctl daemon-reload
I have tried the above unsuccessfully. Even after removing ec2-instance-connect.conf and issuing either systemctl daemon-reload or kill -s HUP <sshd pid> the sshd process is still running using the ec2-instance-connect.conf settings:
sshd: /usr/sbin/sshd -D -o AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys %u %f -o AuthorizedKeysCommandUser ec2-instance-connect [listener] 0 of 10-100 startups
For obvious reasons I am reluctant to tinker more extensively with the sshd configuration on a production server without hearing from the community.
It seems rather questionable (to put it mildly) for a "security update package" to hijack the normal sshd auth process, especially with no well publicized info, only to come to light when I actually have to work on it. The package listing says
Configures ssh daemon to accept EC2 Instance Connect ssh keys
-but what it fails to add is "... and may disable other keys".
We surely cannot be the first ones to encounter this problem??
AWS OFFICIALUpdated 2 years ago
Update: adding a new user (ubuntu
adduser) with a valid public key, and then runningeic_harvest_hostkeysdoes at least allow a brand new user to ssh in to the instance. But the original ami-provided user still does not work, still with the subject error.I will not tag this as 'Answered', since although this helps me to work around the issue, it is not a resolution.