We notices on Monday that ssh connections to our bastion hosts would fail about 80% of the time. If you did connect, your connection would be stable.
Running tcptrack on the bastion host, I seet the failed connections are showing as "reset".
I've tried patching/upgrading to the latest Ubuntu and SSH versions and the problem has persisted.
I have also verified that we are not even getting close to the max number of ports we can have open on the host.
I have confirmed this by trying to connect to the bastion hosts eternal interface from an EC2 instance in our AWS network and my home system with a similar failure rate. Connecting to the bastions internal interface from another EC2 instance does not have this problem.
Bastion OS: Ubuntu18.04 LTS 5.4.0-1066-aws #69~18.04.1-Ubuntu
SSH version: OpenSSH7.6p1-4ubuntu0.6
Anyone know what changed over the weekend and how to fix this problem?