Skip to content

EC2 SSH connection stops working after a period of time while status checks remain 3/3

0

I am experiencing an intermittent SSH connectivity problem with an Amazon EC2 Linux instance.

The instance initially accepts SSH connections normally. I can connect using both Visual Studio Code Remote-SSH and a normal SSH client. The connection remains stable and usable for a period of time.

Then, without any intentional configuration change, the Remote-SSH connection suddenly changes to "Reconnecting to SSH" and eventually fails.

The important part is that after this happens, the problem is not limited to Visual Studio Code.

I also lose access using:

  • Normal SSH from Windows/Git Bash
  • Windows built-in OpenSSH
  • EC2 Instance Connect from the AWS Management Console

The EC2 instance continues to show 3/3 status checks passed in the AWS console.

Connection sequence

  1. EC2 instance is running normally.
  2. SSH connection works successfully.
  3. VS Code Remote-SSH connects successfully.
  4. The connection works normally for some time.
  5. Remote-SSH suddenly changes to "Reconnecting to SSH".
  6. Remote-SSH eventually fails.
  7. Normal SSH connections also fail.
  8. EC2 Instance Connect also fails.
  9. EC2 status checks continue to show 3/3 passed.

SSH troubleshooting

I tested SSH independently of Visual Studio Code.

Git Bash is using:

OpenSSH_10.3p1, OpenSSL 3.5.7

I also tested the Windows built-in OpenSSH client:

OpenSSH_for_Windows_9.5p2, LibreSSL 3.8.2

Both clients can establish the TCP connection to port 22.

For example, the verbose SSH output reaches:

Connecting to <EC2_PUBLIC_IP> port 22. Connection established.

It then reports:

Local version string SSH-2.0-OpenSSH_10.3

However, the SSH session does not continue normally to authentication and shell establishment.

I also tested with:

ssh -i ~/.ssh/tekin.pem -o IdentitiesOnly=yes -vvv ec2-user@<EC2_PUBLIC_IP>

The SSH private key was verified separately using ssh-keygen and appears to be valid.

VS Code

I initially suspected that a VS Code update might be responsible because the problem became noticeable after a VS Code update.

I subsequently downgraded VS Code to version 1.130.0 to test this hypothesis.

The intermittent connectivity problem can still occur with VS Code 1.130.0.

More importantly, when the failure occurs, normal SSH and EC2 Instance Connect also become unavailable. Therefore, the problem does not appear to be limited to VS Code or the Remote-SSH extension.

EC2 status

The instance currently reports:

3/3 status checks passed

Refreshing the EC2 console does not change this status.

Despite the healthy status checks, SSH connectivity can become unavailable.

EC2 Instance Connect

I also attempted:

EC2 Console → Connect → EC2 Instance Connect → Connect

during the connectivity problem.

EC2 Instance Connect also fails.

What I am trying to determine

I would appreciate help identifying the likely root cause.

Could this be related to:

  • sshd becoming unavailable or stuck
  • an operating system resource issue
  • CPU or memory exhaustion
  • disk/filesystem problems
  • EC2 network interface (ENI)
  • VPC/network connectivity
  • Security Group or Network ACL behavior
  • an AWS networking issue
  • an underlying EC2 host issue
  • another condition that would allow EC2 status checks to remain 3/3 while SSH connectivity fails?

The most interesting observation is that TCP port 22 appears reachable initially ("Connection established"), but the SSH protocol negotiation does not complete when the failure occurs.

The problem is intermittent, so exact timestamps may be important for troubleshooting.

I can provide the following if required:

  • full SSH -vvv logs
  • VS Code Remote-SSH logs
  • EC2 screenshots
  • EC2 Instance Connect error screenshots
  • EC2 instance ID
  • AWS Region and Availability Zone
  • exact timestamps of future failures
  • additional diagnostic output from the EC2 instance when access is available

Has anyone experienced a similar situation where SSH, VS Code Remote-SSH, and EC2 Instance Connect all become unavailable while EC2 status checks continue to report 3/3 passed?

5 Answers
0

Hello.

I, too, have experienced an issue in the past where using VS Code Remote-SSH caused a load on the EC2 instance, making SSH connections impossible.
In my case, VSCode Remote SSH was consuming a large amount of memory, causing OOM errors.
The issue was resolved by creating a swap file to increase the temporary memory space and simply upgrading the instance size.
I cannot say for certain without looking at your EC2 instance's syslog, but I suspect there may be a memory-related issue, as indicated here.
https://github.com/microsoft/vscode/issues/175830

EXPERT

answered 25 days ago

EXPERT

reviewed 25 days ago

0

Most likely causes: The sshd process itself has hung, or OS resources are exhausted.

  1. CPU credit exhaustion (for T-series instances): Burstable instances (t2/t3/t4g series) drop to baseline performance when CPU credits run out; this can make forking new processes (i.e., establishing new SSH sessions) extremely slow or effectively unresponsive.

  2. Memory exhaustion: In an OOM (Out of Memory) state, the system cannot spawn new processes; even if the existing sshd listener process remains active, it cannot launch session handlers for new connections. Disk space exhaustion: If partitions like /var or /tmp become full, sshd may fail to write logs or perform PAM-related tasks, leading to similar symptoms.

  3. Factors specific to VS Code Remote-SSH: There are reports that sessions generated by the Remote-SSH extension can accumulate and consume server-side resources over time. This aligns with your observation that the issue became noticeable after a VS Code update.

Most important next step: Use the serial console.

While both SSH and EC2 Instance Connect rely on network stack access, the EC2 serial console provides direct access to the instance's virtual console, bypassing the network stack entirely. This means that even if SSH is completely unresponsive, there is a high probability that you can still log in via the serial console.

answered 25 days ago

EXPERT

reviewed 25 days ago

0

The three usual suspects: CPU is maxed out – Some program is eating 100% of the processor, so "sshd" can't get a turn to respond. Memory is full – The server ran out of RAM. Linux may have killed "sshd" or the system is crawling because it's swapping to disk. Disk is full – The hard drive has zero free space, so "sshd" can't even write log files or create a session. What to do when you get back in: Run "top" to see what's eating CPU/memory Run "df -h" to check disk space Look at "journalctl -u sshd" to see what happened

answered 24 days ago

EXPERT

reviewed 24 days ago

0

Hello AWS Support,

I would like to provide an update regarding the intermittent SSH connectivity issue with my EC2 instance.

The issue currently appears to be resolved or at least no longer reproducible. I have been able to use the instance normally through both VS Code Remote-SSH and standard SSH connections.

During the investigation, I observed the following:

  • The instance is running Amazon Linux 2023.
  • The instance has approximately 913 MiB of total RAM and no swap configured.
  • VS Code Remote-SSH starts VS Code Server processes on the instance. The extension host was using approximately 430–435 MB RSS.
  • CPU load remained very low (approximately 0.00–0.18).
  • Root filesystem usage was approximately 28%, so disk space does not appear to be an issue.
  • I checked the kernel/journal logs for OOM (Out Of Memory), "killed process", and related memory-pressure messages. No OOM events were found.
  • I tested file creation, reading, and deletion through the Remote-SSH terminal successfully.
  • I disconnected and reconnected using VS Code Remote-SSH successfully. The previous VS Code Extension Host process was terminated and a new one was created; I did not observe multiple Extension Host processes accumulating.
  • I also established a separate standard SSH connection while the VS Code Remote-SSH connection was active. Both connections worked simultaneously.
  • The sshd process and multiple sshd sessions were functioning normally.

I also discovered that the user's ~/.bashrc file contained a corrupted PS1 prompt definition and an unexpected "OA" line. This was causing shell errors such as:

-bash: 5: command not found -bash: 82m]u[e[0m]@[e[38: command not found

I corrected the ~/.bashrc configuration, and these login/shell errors no longer occur.

At this point, the instance is stable and I can connect through both normal SSH and VS Code Remote-SSH without problems.

However, I would still like AWS Support to help determine the root cause of the original incident. Previously, after using VS Code Remote-SSH, the EC2 instance eventually became inaccessible through SSH, and I was also unable to connect using EC2 Instance Connect. Rebooting the instance was required before connectivity could be restored.

Could you please review the available EC2/system-level logs and determine whether there was any evidence of:

  1. SSH/sshd failure or resource exhaustion,
  2. memory pressure or process termination,
  3. network-level connectivity problems,
  4. EC2 host or instance-level issues,
  5. or any interaction between VS Code Remote-SSH / VS Code Server and the observed SSH connectivity failure?

The issue is currently not reproducible, so I would especially appreciate any analysis of the historical period when the instance became completely inaccessible.

Thank you.

answered 23 days ago

0

The 'works fine then stops after a while' pattern with status checks still green is almost always the Linux OOM killer taking out sshd. EC2 status checks only measure CPU and network reachability, not memory, so a box can be quietly running out of RAM and look perfectly healthy in the console.

Quick thing to check right now: run dmesg | grep -i oom on the instance and look for lines mentioning sshd. If you see them, that's your answer.

Also worth enabling AWS Systems Manager Session Manager as a fallback if you haven't already. It's free and gets you a shell even when SSH is completely dead, which is really useful for diagnosing the next occurrence live.

If dmesg confirms OOM, the fix is usually adding swap (EC2 doesn't configure any by default) and installing the CloudWatch Agent to start collecting memory metrics so you can actually see the pressure building.

I've put together a structured write-up covering the root-cause matrix for exactly this failure pattern, with diagnostic commands, a CloudWatch Agent config for memory and sshd process monitoring, and an optional auto-remediation setup. Happy to share it. One quick question that would help me point you to the right section first: what error do you see on the client when SSH fails, 'Connection refused' or 'Connection timed out'? That single detail tells us whether sshd is dead or the instance is fully unresponsive, and changes what to do next.

answered 22 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.