How to solve the problem of connection failure after EC2 instance Ubuntu upgrade

Question

My Ubuntu instance is 18.04. When I ran "sudo do-release-upgrade" to upgrade to Ubuntu20.04, it was normal. Other services were also normal. When I upgraded from Ubuntu20.04 to Ubuntu22.04 again, I used winscp and Putty cannot connect to the instance

This is the AWS console instance system log
[[0;32m  OK  [0m] Finished [0;1;39mInitial cloud-iniâ¦ob (metadata service crawler)[0m.
[[0;32m  OK  [0m] Reached target [0;1;39mCloud-config availability[0m.
[[0;32m  OK  [0m] Reached target [0;1;39mNetwork is Online[0m.
[[0;32m  OK  [0m] Reached target [0;1;39mSystem Initialization[0m.
[[0;32m  OK  [0m] Started [0;1;39mACPI Events Check[0m.
[[0;32m  OK  [0m] Started [0;1;39mresolvconf-pull-resolved.path[0m.
[[0;32m  OK  [0m] Started [0;1;39mDaily apt download activities[0m.
[[0;32m  OK  [0m] Started [0;1;39mDaily apt upgrade and clean activities[0m.
[[0;32m  OK  [0m] Started [0;1;39mDaily dpkg database backup timer[0m.
[[0;32m  OK  [0m] Started [0;1;39mPeriodic ext4 Onliâ¦ata Check for All Filesystems[0m.
[[0;32m  OK  [0m] Started [0;1;39mDiscard unused blocks once a week[0m.
[[0;32m  OK  [0m] Started [0;1;39mRefresh fwupd metadata regularly[0m.
[[0;32m  OK  [0m] Started [0;1;39mDaily rotation of log files[0m.
[[0;32m  OK  [0m] Started [0;1;39mDaily man-db regeneration[0m.
[[0;32m  OK  [0m] Started [0;1;39mMessage of the Day[0m.
[[0;32m  OK  [0m] Started [0;1;39mUpdate the plocate database daily[0m.
[[0;32m  OK  [0m] Started [0;1;39mDaily Cleanup of Temporary Directories[0m.
[[0;32m  OK  [0m] Started [0;1;39mDownload data for â¦ailed at package install time[0m.
[[0;32m  OK  [0m] Started [0;1;39mCheck to see whethâ¦w version of Ubuntu available[0m.
[[0;32m  OK  [0m] Reached target [0;1;39mPath Units[0m.
[[0;32m  OK  [0m] Reached target [0;1;39mTimer Units[0m.
[[0;32m  OK  [0m] Listening on [0;1;39mACPID Listen Socket[0m.
[[0;32m  OK  [0m] Listening on [0;1;39mcloud-init hotplug hook socket[0m.
[[0;32m  OK  [0m] Listening on [0;1;39mD-Bus System Message Bus Socket[0m.
         Starting [0;1;39mDocker Socket for the API[0m...
[[0;32m  OK  [0m] Listening on [0;1;39mOpen-iSCSI iscsid Socket[0m.
[[0;32m  OK  [0m] Listening on [0;1;39mSocket unix for snap application lxd.daemon[0m.
         Starting [0;1;39mSocket activation for snappy daemon[0m...
[[0;32m  OK  [0m] Listening on [0;1;39mUUID daemon activation socket[0m.
[[0;32m  OK  [0m] Reached target [0;1;39mPreparation for Remote File Systems[0m.
[[0;32m  OK  [0m] Reached target [0;1;39mRemote File Systems[0m.
[[0;32m  OK  [0m] Finished [0;1;39mAvailability of block devices[0m.
[[0;32m  OK  [0m] Listening on [0;1;39mDocker Socket for the API[0m.
[[0;32m  OK  [0m] Listening on [0;1;39mSocket activation for snappy daemon[0m.
[[0;32m  OK  [0m] Reached target [0;1;39mSocket Units[0m.
[[0;32m  OK  [0m] Reached target [0;1;39mBasic System[0m.
         Starting [0;1;39mAccounts Service[0m...
[[0;32m  OK  [0m] Started [0;1;39mAmazon CloudWatch Agent[0m.
         Starting [0;1;39mLSB: automatic crash report generation[0m...
         Starting [0;1;39mDeferred execution scheduler[0m...
         Starting [0;1;39mLSB: Daemon for AWSLogs agent.[0m...
[[0;32m  OK  [0m] Started [0;1;39mClamAV virus database updater[0m.
         Starting [0;1;39mcontainerd container runtime[0m...
[[0;32m  OK  [0m] Started [0;1;39mRegular background program processing daemon[0m.
[[0;32m  OK  [0m] Started [0;1;39mD-Bus System Message Bus[0m.
[[0;32m  OK  [0m] Started [0;1;39mSave initial kernel messages after boot[0m.
         Starting [0;1;39mRemove Stale Onliâ¦t4 Metadata Check Snapshots[0m...
         Starting [0;1;39mRecord successful boot for GRUB[0m...
[[0;32m  OK  [0m] Started [0;1;39mirqbalance daemon[0m.
[[0;32m  OK  [0m] Started [0;1;39mFUSE filesystem for LXC[0m.
         Starting [0;1;39mDispatcher daemon for systemd-networkd[0m...
         Starting [0;1;39mAuthorization Manager[0m...
         Starting [0;1;39mPostfix Mail Transport Agent (instance -)[0m...
[[0;32m  OK  [0m] Started [0;1;39mQualys cloud agent daemon[0m.
         Starting [0;1;39mresolvconf-pull-resolved.service[0m...
         Starting [0;1;39mSystem Logging Service[0m...
         Starting [0;1;39mLSB: OpenLDAP staâ¦ Directory Access Protocol)[0m...
         Starting [0;1;39mService for snap application lxd.activate[0m...
[[0;32m  OK  [0m] Reached target [0;1;39mPreparation for Logins[0m.
         Starting [0;1;39mSnap Daemon[0m...
         Starting [0;1;39mOpenBSD Secure Shell server[0m...
         Starting [0;1;39mUser Login Management[0m...
         Starting [0;1;39mDisk Manager[0m...
[[0;32m  OK  [0m] Started [0;1;39mDeferred execution scheduler[0m.
[[0;32m  OK  [0m] Started [0;1;39mSystem Logging Service[0m.
[[0;32m  OK  [0m] Started [0;1;39mLSB: automatic crash report generation[0m.
[[0;32m  OK  [0m] Finished [0;1;39mRecord successful boot for GRUB[0m.
[[0;32m  OK  [0m] Started [0;1;39mLSB: Daemon for AWSLogs agent.[0m.
         Starting [0;1;39mGRUB failed boot detection[0m...
[[0;32m  OK  [0m] Finished [0;1;39mGRUB failed boot detection[0m.
[[0;32m  OK  [0m] Started [0;1;39mAuthorization Manager[0m.
         Starting [0;1;39mModem Manager[0m...
[[0;32m  OK  [0m] Started [0;1;39mOpenBSD Secure Shell server[0m.
[[0;32m  OK  [0m] Finished [0;1;39mRemove Stale Onliâ¦ext4 Metadata Check Snapshots[0m.
[[0;32m  OK  [0m] Started [0;1;39mUser Login Management[0m.
[[0;32m  OK  [0m] Started [0;1;39mUnattended Upgrades Shutdown[0m.
[[0;32m  OK  [0m] Started [0;1;39mDispatcher daemon for systemd-networkd[0m.
[[0;32m  OK  [0m] Started [0;1;39mLSB: OpenLDAP stanâ¦ht Directory Access Protocol)[0m.
[[0;32m  OK  [0m] Started [0;1;39mDisk Manager[0m.
[[0;32m  OK  [0m] Started [0;1;39mAccounts Service[0m.
[[0;32m  OK  [0m] Started [0;1;39mModem Manager[0m.
[[0;32m  OK  [0m] Finished [0;1;39mresolvconf-pull-resolved.service[0m.
[[0;32m  OK  [0m] Started [0;1;39mcontainerd container runtime[0m.
         Starting [0;1;39mDocker Application Container Engine[0m...
[[0;32m  OK  [0m] Started [0;1;39mPostfix Mail Transport Agent (instance -)[0m.
         Starting [0;1;39mPostfix Mail Transport Agent[0m...
[[0;32m  OK  [0m] Finished [0;1;39mPostfix Mail Transport Agent[0m.
[ [0;31m*[0;1;31m*[0m[0;31m*  [0m] (2 of 3) A start job is running forâ¦ Container Engine (10s / no limit)
M
[K[[0;32m  OK  [0m] Started [0;1;39mSnap Daemon[0m.
[K         Starting [0;1;39mWait until snapd is fully seeded[0m...
         Starting [0;1;39mTime & Date Service[0m...
[[0;32m  OK  [0m] Started [0;1;39mTime & Date Service[0m.
[[0;32m  OK  [0m] Finished [0;1;39mWait until snapd is fully seeded[0m.
         Starting [0;1;39mApply the settings specified in cloud-config[0m...
[   16.893996] cloud-init[2179]: Cloud-init v. 23.3.1-0ubuntu1~22.04.1 running 'modules:config' at Thu, 19 Oct 2023 06:37:29 +0000. Up 16.84 seconds.
[[0;32m  OK  [0m] Finished [0;1;39mApply the settings specified in cloud-config[0m.
         Starting [0;1;39mPermit User Sessions[0m...
[[0;32m  OK  [0m] Finished [0;1;39mPermit User Sessions[0m.
         Starting [0;1;39mHold until boot process finishes up[0m...
         Starting [0;1;39mTerminate Plymouth Boot Screen[0m...
[[0;32m  OK  [0m] Finished [0;1;39mHold until boot process finishes up[0m.
[[0;32m  OK  [0m] Started [0;1;39mSerial Getty on ttyS0[0m.
         Starting [0;1;39mSet console scheme[0m...
[[0;32m  OK  [0m] Finished [0;1;39mSet console scheme[0m.
[[0;32m  OK  [0m] Created slice [0;1;39mSlice /system/getty[0m.
[[0;32m  OK  [0m] Started [0;1;39mGetty on tty1[0m.
[[0;32m  OK  [0m] Reached target [0;1;39mLogin Prompts[0m.
[[0;32m  OK  [0m] Finished [0;1;39mTerminate Plymouth Boot Screen[0m.
[[0;32m  OK  [0m] Started [0;1;39mlibcontainer contaâ¦b859fc96d2ea413d79883d306aab5[0m.
[   18.835068] systemd[1]: Failed to create symlink /sys/fs/cgroup/net_cls: Operation not permitted

Ubuntu 22.04.3 LTS ip-172-31-1-19 ttyS0

ip-172-31-1-19 login: [   18.939311] cloud-init[2892]: Cloud-init v. 23.3.1-0ubuntu1~22.04.1 running 'modules:final' at Thu, 19 Oct 2023 06:37:31 +0000. Up 18.89 seconds.
[   19.011435] cloud-init[2892]: Cloud-init v. 23.3.1-0ubuntu1~22.04.1 finished at Thu, 19 Oct 2023 06:37:31 +0000. Datasource DataSourceEc2Local.  Up 19.00 seconds

Answer

OpenSSH introduced a change in version 8.9p1 related to the MaxStartups configuration item. In previous versions, MaxStartups allowed larger values (such as 4096), but in the new version, due to the switch to using the ppoll(2) system call, the maximum value of MaxStartups is limited to 1023.
This is a change in OpenSSH to accommodate the new ppoll(2) behavior. Since this change may affect old configurations, it is recommended to ensure that the MaxStartups configuration item in your sshd_config file does not exceed 1023. If a larger value is set in your configuration, it should be modified to 1023 or smaller to ensure OpenSSH operates properly.
You mentioned some suggestions for possible improvements, such as updating documentation, sshd -t showing errors when checking configuration, or automatically limiting MaxStartups to 1023. These suggestions can be submitted to the OpenSSH community to help improve the software's user experience and documentation accuracy.
In summary, make sure your MaxStartups configuration does not exceed 1023 and adhere to OpenSSH's new behavior rules to ensure that your SSH server operates properly. If you set a larger value previously, make sure to change it accordingly in the new version.

Answer

Hello,

I understand that you are unable to access the ubuntu instance after upgrade to 22.04 via putty or wincp.

I was able to replicate the exact issue on my test environment and was getting below error.

```
No support authentication methods available (server sent: publickey)
```

Further I did some research on this issue with 22.04 version of ubuntu and found that “ssh-rsa” is now disabled by default in latest version of SSH(OpenSSH_8.9p1) which is installed on the latest Ubuntu versions. You can see the bug **1961833** to learn how to selectively re-enable it if necessary.

[+]https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1961833

According to the above link you can add below lines in **/etc/ssh/sshd_config** file and restart SSH service, which will enable RSA/SHA1 for host:-

```
HostkeyAlgorithms +ssh-rsa
PubkeyAcceptedAlgorithms +ssh-rsa
```

Please see the snippet for reference:

```
grep -i ssh-rsa /etc/ssh/sshd_config
HostkeyAlgorithms +ssh-rsa
PubkeyAcceptedAlgorithms +ssh-rsa
```

**Note : You could update the SSH configuration with above details just before performing reboot after upgrade from 20.04 to 22.04, to avoid connection issue.**

Alternatively you can connect to your instances using the sessions manager service to make above changes as the “EC2 instance connect” option is also impacted due to this issue and you will not be able to connect to Ubuntu 22.04 version instance using “EC2 instance connect”. To connect to Ec2 instance using Sessions manager you can follow steps in below link:

[+]. https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started.html

[+]. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/session-manager.html

By default SSM agent should be running on the instance, so next step to use SSM is to attach SSM role to the instance. First of all you need to attach AmazonSSMManagedInstanceCore policy to the instance so that Systems Manager can access the instance.

```
[1] Let's navigate to IAM service and create a role.

[2] Choose the service as EC2

[3] You need to attach ‘AmazonSSMManagedInstanceCore’ policy to the role.

[4] Give a name to the role and finally create the role.

[5] Then you need to attach the role to your instance, by right click on the instance >> select security >> Modify IAM role >> select newly created role >> click update IAM role

[6]Click on the instance and then click connect >> select session manager >> click connect 
```

**Note : If the connect option is not present after attaching the above IAM role, please wait for 5-10 minutes and also try rebooting the server**

Hope this helps. Please do let me know if you have any further queries or concerns. Have a great day ahead!!

How to solve the problem of connection failure after EC2 instance Ubuntu upgrade

Relevant content