EC2 instance unaccessible but does ping

Question

My EC2 instance started behaving odd about two weeks ago, whereas it ran without much problem 3+ years before.  The instance can be pinged, but the apache website on it is unavailable,  and a remote ssh session cannot be established.  Using the AWS EC2 web tools trying to connect with a terminal or console session is also not possible.  I have a status check setup to send an alarm when the status check fails, but apparently it does not fail because the instance is pingable.

The workaround right now is stopping the instance, waiting a few (10) seconds and restarting it again.  I also tried just a plain reboot through the AWS tools, but this fails, so I really need to stop, wait and start it before it becomes available again.  After that it will run a few days before the problem starts occurring again.
 
I am at my wit’s ends attempting to troubleshoot this issue, as I find nothing out of the ordinary in the logs of the Ubuntu 20.04.6 LTS server on this instance.  I am hoping someone here can point me in the right direction to figure out this issue.

I hope I'm providing enough information, this is my first visit to this forum, so forgive my ignorance.  Let me know what other information may be needed.

Accepted Answer

After doing some more research into this problem, I found out that the EBS read/write on this instance are way higher than they should be. The web site should be visited by no more than a few people a week (family and friends), but appears to be hammered by crawlers, robots, spammers and hackers. If this goes on Lightsail will probably be a better solution. Too bad as I also use this instance to keep my AWS skills to par.

For now I will try to do some mitigation with anti-spam and bot blockers. No sense in upgrading to a t4g.small for something that should be this small and low in traffic.

For now I will try to do some mitigation with anti-spam and bot blockers.  No sense in upgrading to a t4g.small for something that should be this small and low in traffic.

Answer

Sounds like this may be an Ubuntu issue.

I have a couple of suggestions to help troubleshoot further.

[https://serverfault.com/questions/1030527/exiting-ssh-sessions-suddenly-killing-apache.
](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html)

I generally don't like to use ssh to connect to my instances.  Using Systems Manager Session Manager on the host may help you get into the system, while it is up and not responding to ssh, to look around and find more info.

https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html

Question . . .
**Have you patched recently?**

Answer

Could it be resource exhaustion? Take a look in the Monitoring tab of AWS Console, and look for anything that's unusually high, or otherwise out-of-line.

Also consider setting up CloudWatch agent to collect more detailed system logs https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html

What instance type is being used here?

Answer

Anything is possible, but I don't see anything that pops out on the monitoring tab.  It was hung again this morning, so far not been possible to connect a terminal session.  If that were possible, I could do some better troubleshooting...

I guess I can delve more into the CLoudWatch agent.  That will take me a while to research and setup.  I've been quite intimidated by the AWS setup of tools. :-)

The instance is a t2.small

Answer

:-)  I patch regularly, usually weekly.

This morning when it didn't respond, I tried to use session manager to connect instead, but unfortunately that did not work either.   I found a console option too, but also would not connect.  Can't find that console option anymore, I should have bookmarked it.

I will take a look at the link you posted.

Edit: It looks like the ssh problem mentioned in the link was not applicable.  That occured on Ubuntu 16.04.  I am running 20.04 LTS and also the apache session is started as root under which this should not occur

EC2 instance unaccessible but does ping

Relevant content