Cannot consistently ssh into EC2 instance

0

I'm having trouble consistently connecting via SSH to my EC2 instance. I'll be able to connect for a period of time, then I'll randomly get disconnected, be unable to connect for a bit, and over time, the issue will resolve itself.

Any ideas of things to try to make it so that I can consistently ssh into the server? So far I've tried:

  • Restarting the instance and completely creating a new EC2 instance (with Debian rather than AWS Linux)
  • Connecting from a different machine (also just times out)
  • Connecting from CloudShell (I just get the following): Enter image description here

Here's the output that I get during the periods of being unable to connect:

[20:21:40.269] Log Level: 2
[20:21:40.279] VS Code version: 1.87.2
[20:21:40.280] Remote-SSH version: remote-ssh@0.109.0
[20:21:40.280] linux x64
[20:21:40.285] SSH Resolver called for "ssh-remote+ec2-x-x-x-x.us-east-2.compute.amazonaws.com", attempt 1
[20:21:40.285] "remote.SSH.useLocalServer": true
[20:21:40.285] "remote.SSH.useExecServer": true
[20:21:40.286] "remote.SSH.path": undefined
[20:21:40.286] "remote.SSH.configFile": undefined
[20:21:40.286] "remote.SSH.useFlock": true
[20:21:40.286] "remote.SSH.lockfilesInTmp": false
[20:21:40.286] "remote.SSH.localServerDownload": auto
[20:21:40.286] "remote.SSH.remoteServerListenOnSocket": false
[20:21:40.286] "remote.SSH.showLoginTerminal": false
[20:21:40.286] "remote.SSH.defaultExtensions": []
[20:21:40.286] "remote.SSH.loglevel": 2
[20:21:40.288] "remote.SSH.enableDynamicForwarding": true
[20:21:40.288] "remote.SSH.enableRemoteCommand": false
[20:21:40.288] "remote.SSH.serverPickPortsFromRange": {}
[20:21:40.288] "remote.SSH.serverInstallPath": {}
[20:21:40.293] SSH Resolver called for host: ec2-x-x-x-x.us-east-2.compute.amazonaws.com
[20:21:40.293] Setting up SSH remote "ec2-x-x-x-x.us-east-2.compute.amazonaws.com"
[20:21:40.296] Acquiring local install lock: /tmp/vscode-remote-ssh-a0a85fa7-install.lock
[20:21:40.298] Looking for existing server data file at /home/user/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-a0a85fa7-863d2581ecda6849923a2118d93a088b0745d9d6-0.109.0-es/data.json
[20:21:40.300] Using commit id "863d2581ecda6849923a2118d93a088b0745d9d6" and quality "stable" for server
[20:21:40.304] Install and start server if needed
[20:21:40.308] PATH: /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
[20:21:40.308] Checking ssh with "ssh -V"
[20:21:40.316] > OpenSSH_9.4p1 Debian-1, OpenSSL 3.0.12 24 Oct 2023

[20:21:40.318] askpass server listening on /run/user/1000/vscode-ssh-askpass-bc66b0fd8a9741d69bc5df6f1fc02af460bfb623.sock
[20:21:40.319] Spawning local server with {"serverId":1,"ipcHandlePath":"/run/user/1000/vscode-ssh-askpass-c563d75a23e6fcadf194436aee5e4f470a48b5f7.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","37511","-o","ConnectTimeout=15","ec2-x-x-x-x.us-east-2.compute.amazonaws.com"],"serverDataFolderName":".vscode-server","dataFilePath":"/home/user/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-a0a85fa7-863d2581ecda6849923a2118d93a088b0745d9d6-0.109.0-es/data.json"}
[20:21:40.319] Local server env: {"SHELL":"/bin/bash","DISPLAY":":0","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.109.0/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/usr/share/code/code","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"","VSCODE_SSH_ASKPASS_MAIN":"/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.109.0/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/run/user/1000/vscode-ssh-askpass-bc66b0fd8a9741d69bc5df6f1fc02af460bfb623.sock"}
[20:21:40.323] Spawned 2073
[20:21:40.449] > local-server-1> Running ssh connection command: ssh -v -T -D 37511 -o ConnectTimeout=15 ec2-x-x-x-x.us-east-2.compute.amazonaws.com
[20:21:40.451] > local-server-1> Spawned ssh, pid=2081
[20:21:40.455] stderr> OpenSSH_9.4p1 Debian-1, OpenSSL 3.0.12 24 Oct 2023
[20:21:55.475] stderr> Connection timed out during banner exchange
[20:21:55.475] stderr> Connection to 3.139.71.64 port 22 timed out
[20:21:55.476] > local-server-1> ssh child died, shutting down
[20:21:55.482] Local server exit: 0
[20:21:55.482] Received install output: local-server-1> Running ssh connection command: ssh -v -T -D 37511 -o ConnectTimeout=15 ec2-x-x-x-x.us-east-2.compute.amazonaws.com
local-server-1> Spawned ssh, pid=2081
OpenSSH_9.4p1 Debian-1, OpenSSL 3.0.12 24 Oct 2023
Connection timed out during banner exchange
Connection to 3.139.71.64 port 22 timed out
local-server-1> ssh child died, shutting down

[20:21:55.486] Resolver error: Error: The connection timed out
	at g.Offline (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.109.0/out/extension.js:2:499785)
	at /home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.109.0/out/extension.js:2:495810
	at t.handleInstallOutput (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.109.0/out/extension.js:2:496125)
	at e (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.109.0/out/extension.js:2:558185)
	at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
	at async /home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.109.0/out/extension.js:2:580094
	at async t.withShowDetailsEvent (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.109.0/out/extension.js:2:583753)
	at async /home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.109.0/out/extension.js:2:554890
	at async T (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.109.0/out/extension.js:2:552941)
	at async t.resolveWithLocalServer (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.109.0/out/extension.js:2:554430)
	at async k (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.109.0/out/extension.js:2:577311)
	at async t.resolve (/home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.109.0/out/extension.js:2:581124)
	at async /home/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.109.0/out/extension.js:2:847283
[20:21:55.491] ------

It's so inconsistent, and I have no idea how to start troubleshooting. Any ideas would be super appreciated.

  • Based on log you sent, your instance needs more memory. The solution is change your instance type or create and activate a swap file.

    In my experience, usually, a 1GB swapfile is sufficient for save us of pay more for a temporary peak of use.

asked a month ago193 views
2 Answers
3
Accepted Answer

Hello.

It seems that a timeout error is occurring, so I think it is necessary to check the EC2 OS side system log and SSH log instead of the client side log.
Please check the system logs and SSH logs during the time period when connection was not possible.
It may also be a good idea to check the load status of EC2 during that time period.

profile picture
EXPERT
answered a month ago
profile picture
EXPERT
reviewed a month ago
profile picture
EXPERT
reviewed a month ago
profile picture
EXPERT
shibata
reviewed a month ago
  • As per Riku_Kobayashi's answer, I think that the EC2 side is unable to accept SSH connections due to high load. Please check the CPU and memory usage of EC2.

    As a temporary solution, I recommend upgrading the EC2 instance type.

  • After rebooting the instance, everything worked fine for ~5 minutes, then my connections dropped. At the time, the server wasn't running any non-base processes, and I was just editing files over ssh.

    Here's the EC2 monitoring info: https://imgur.com/a/STsPOfF System log file: https://pastebin.com/b5rE3PDa

    What should I look for to see if it's a CPU and memory usage issue? Is there some way to do a flush of the memory in case previously caching is causing issues? I guess I'm finding it strange that the instance immediately reaches memory / CPU problems upon startup when no non-standard process are triggered to run

  • Yep - seems that it was a CPU/memory usage problem that is caused due to VSCode extensions (per this thread: https://github.com/microsoft/vscode-remote-release/issues/2692 )

    Appreciate everyone's help!

  • If you don't have enough memory, I think it's a good idea to create a swap space as described in the document below. https://repost.aws/knowledge-center/ec2-memory-swap-file

1

Hi.

oom-kill - some process "eating" all you memory, maybe add more swap to this EC2 to keep and running little longer and observe process. Looke like "node" process, nodejs?

Thanks,

profile picture
answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions