"Permission denied" even though "ec2-instance-connect send-serial-console-ssh-public-key" succeeds

Question

I'm writing a script to SSH into an EC2 instance serial console. I followed the instructions/example in the [CLI reference](https://docs.aws.amazon.com/cli/latest/reference/ec2-instance-connect/send-serial-console-ssh-public-key.html), but even though my `ec2-instance-connect send-serial-console-ssh-public-key`
command succeeds, the subsequent `ssh` attempt always gets "Permission denied".

I've checked that I'm using the supported key sizes and algorithm, I've checked the SSH options, and I've searched through this forum, but have not found anything else to try. I am able to reach the EC2 instance serial console via the Web interface, and the instance is up and running.

Below is the debug output of my script, showing the construction of the SSH key, transmitting it to AWS, and then attempting the SSH connection. Can anyone suggest what might be wrong or have hints on how to troubleshoot further?
```
[user@manager-host ~]$ bash -x ~/asgconsole us-east-2 i-0729a2db5e79493db
+ USAGE='Usage: asgconsole REGION INSTANCE-NAME'
+ set -u
+ set -e
+ REGION=us-east-2
+ INSTANCE=i-0729a2db5e79493db
+ trap 'echo; echo ERROR trying to $TASK: $?; echo; exit 23' EXIT
+ TASK='create temp directory'
++ mktemp -d /tmp/asgconsole-user-XX
+ TMPDIR=/tmp/asgconsole-user-bnOBWWuQVw
+ SSH_OPTS='-F /dev/null -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes'
+ set -o
+ grep -q 'xtrace[ 	]*on'
+ SSH_OPTS='-v -F /dev/null -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes'
+ TASK='generate an SSH key to load into the console'
+ ssh-keygen -b 2048 -q -t RSA -f /tmp/asgconsole-user-bnOBWWuQVw/conkey -N ''
+ TASK='confirm SSH key pair created properly'
+ SSH_PUBKEY=/tmp/asgconsole-user-bnOBWWuQVw/conkey.pub
+ SSH_KEY=/tmp/asgconsole-user-bnOBWWuQVw/conkey
+ [[ -s /tmp/asgconsole-user-bnOBWWuQVw/conkey ]]
+ [[ -s /tmp/asgconsole-user-bnOBWWuQVw/conkey.pub ]]
+ TASK='send SSH public key to AWS for console access'
+ aws ec2-instance-connect send-serial-console-ssh-public-key --region us-east-2 --instance-id i-0729a2db5e79493db --ssh-public-key file:///tmp/asgconsole-user-bnOBWWuQVw/conkey.pub
{
    "RequestId": "c0c75994-a500-4b11-b266-cb6ab86ba675",
    "Success": true
}
+ TASK='sshing into EC2 instance i-0729a2db5e79493db serial console'
+ echo 'sshing into EC2 instance i-0729a2db5e79493db serial console...'
sshing into EC2 instance i-0729a2db5e79493db serial console...
+ ssh -v -F /dev/null -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -i /tmp/asgconsole-user-bnOBWWuQVw/conkey i-0729a2db5e79493db@serial-console.ec2-instance-connect.us-east-2.aws
OpenSSH_8.7p1, OpenSSL 3.0.8 7 Feb 2023
debug1: Reading configuration data /dev/null
debug1: Connecting to serial-console.ec2-instance-connect.us-east-2.aws [3.17.136.3] port 22.
debug1: Connection established.
debug1: identity file /tmp/asgconsole-user-bnOBWWuQVw/conkey type 0
debug1: identity file /tmp/asgconsole-user-bnOBWWuQVw/conkey-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.7
debug1: Remote protocol version 2.0, remote software version AWS_1.0
debug1: compat_banner: no match: AWS_1.0
debug1: Authenticating to serial-console.ec2-instance-connect.us-east-2.aws:22 as 'i-0729a2db5e79493db'
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: ecdh-sha2-nistp256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: aes128-gcm@openssh.com MAC:  compression: none
debug1: kex: client->server cipher: aes128-gcm@openssh.com MAC:  compression: none
debug1: kex: ecdh-sha2-nistp256 need=16 dh_need=16
debug1: kex: ecdh-sha2-nistp256 need=16 dh_need=16
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-rsa SHA256:EhwPkTzRtTY7TRSzz26XbB0/HvV9jRM7mCZN0xw/d/0
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
Warning: Permanently added 'serial-console.ec2-instance-connect.us-east-2.aws' (RSA) to the list of known hosts.
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /tmp/asgconsole-user-bnOBWWuQVw/conkey RSA SHA256:w48qPtnvk0svNjHWq3KiN0S7Q33YGRuyE9wDfC+TNpA explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=
debug1: kex_input_ext_info: ping@openssh.com (unrecognised)
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /tmp/asgconsole-user-bnOBWWuQVw/conkey RSA SHA256:w48qPtnvk0svNjHWq3KiN0S7Q33YGRuyE9wDfC+TNpA explicit
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
i-0729a2db5e79493db@serial-console.ec2-instance-connect.us-east-2.aws: Permission denied (publickey).
```

cloudconfuser · Accepted Answer

The problem turned out to be in the SSH command - the instance name needs to have the string ".port0" appended to it. Once that was fixed everything worked as documented.

Varunprakash Shanmugam · Answer

The "Permission denied" error you’re encountering is a common issue when using the EC2 instance serial console with SSH, particularly after successfully uploading the public key but failing the SSH authentication. Here are a few troubleshooting steps that might resolve this issue:

1. **Confirm the Username**: Double-check that you’re using the correct username when connecting via SSH. For Amazon Linux or Amazon Linux 2, use `ec2-user`. If it’s Ubuntu, use `ubuntu`, and for other distributions, refer to [the list of default usernames](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connection-prereqs.html#connection-prereqs-get-default-user).

2. **Key Transmission Timing**: Sometimes, there’s a slight delay between sending the SSH key and when it becomes accessible on the EC2 instance. Try waiting a few seconds after the `send-serial-console-ssh-public-key` command completes before initiating the SSH connection.

3. **Verify Instance IAM Role Permissions**: Ensure the instance’s IAM role includes the `ec2-instance-connect:SendSerialConsoleSSHPublicKey` permission. Even though the command succeeded, permission inconsistencies can sometimes block access at later steps.

4. **Double-Check SSH Key Format**: Ensure your key follows the OpenSSH RSA format. If using RSA, some instances may require specific formats, so make sure the key generation command (`ssh-keygen -t rsa -b 2048`) is correctly creating a compatible key type.

5. **Inspect Serial Console Access Settings**: In the AWS Console, under **EC2 > Instances > Actions > Monitor and troubleshoot > Manage console access**, ensure your EC2 instance serial console is enabled.

6. **Add Debugging Flags to SSH Command**: To get more detailed logs, consider adding `-vvv` to your SSH command, which might provide further insight into why authentication fails.

Try these adjustments, and let me know if you’re still seeing the "Permission denied" message after each step.

"Permission denied" even though "ec2-instance-connect send-serial-console-ssh-public-key" succeeds

Relevant content