ssh key-pair anomaly on copying AMI

Question

I made an image of an instance running in N. Virginia, copied it to Singapore and launched it. On launch, it told me that I have no existing key-pairs, so I had to create a new one. That seemed odd, as I had a key-pair for the instance running in N. Virginia.

After launch, when I tried to connect via SSH I got 
```
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
```
After trying several solutions, I tried connecting to the new, Singapore instance using the key from the old N. Virginia instance. I connected successfully.  I checked the authorized_keys file and saw that it contained the old N. Virginia key.

I extracted public key from the new Singapore key-pair that I had downloaded and concatenated that into the authorized_keys file on the new Singapore instance.

When I tried to connect using the new key_pair, I again got  
```
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
```

The only way that I can connect to the new instance is by using the old key-pair. It's not too much of a problem as I'd have chosen to use that key-pair anyway, had I been given a choice when I launched the instance for the first time.

However, I'm concerned that it won't accept the new key-pair, even though I've added it to authorized keys. I'm also surprised that the process for copying an AMI has what appears to be a bug.

Answer

Hi!  When you created the AMI, did you remove the SSH host key pairs located in /etc/ssh?  If you do so, this will force SSH to generate new and uique SSH key pairs when another instance is launched using the AMI (such as what you did in the Singapore region).

This includes the following key files that may be present:

* ssh_host_dsa_key
* ssh_host_dsa_key.pub
* ssh_host_key
* ssh_host_key.pub
* ssh_host_rsa_key
* ssh_host_rsa_key.pub
* ssh_host_ecdsa_key
* ssh_host_ecdsa_key.pub
* ssh_host_ed25519_key
* ssh_host_ed25519_key.pub

If you remove the SSH host key pairs prior to creating the AMI, then the shared AMI should use the new keys as you expected to happen.   You can use something like shred to remove files: https://www.gnu.org/software/coreutils/manual/html_node/shred-invocation.html.

Edit: This is documented behavior.  Check out the Remove SSH Key Pairs section of https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/building-shared-amis.html for a security practice to use when sharing AMIs.  I would recommend removing the SSH key pairs when you're creating the image and this issue should resolve when you start a new instance from that new image.

Answer

This is expected behaviour.

In the console, key-pairs are region-specific - creating a key-pair in one region does not make it available in other regions. You can import key-pairs so that you have the same key-pair in each region; but creating a new key-pair in the Singapore region (even with the same name) is actually creating a new key-pair.

When you copied the AMI from N. Virginia to Singapore the AMI contained the public key for the key-pair from the N. Virginia region. Had you created a new AMI in the Singapore region it would have used the key material from the key-pair created in the Singapore region; but because it was copied it contained the original key material.

In the end, you got to a valid solution - taking the Singapore public key material and putting it on the instance; but you might also have created a key-pair and imported it into both regions: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-ssh-key-pair-regions/

ssh key-pair anomaly on copying AMI

Relevant content