ec2 instance loses network connectivity

Question

I have a web service consisting of two EC2 instances running Tomcat behind a load balancer. Each instance is t2.large.

About once a week, one of the instances stops responding, and it's always the same one. When I reboot it from the EC2 console then it returns to normal. Examination of the logs shows that the instance was running but could not access the internet during the outage. The logs show network timeouts trying to connect to various AWS services.

CPU usage, which is normally less than 10%, spikes up to about 60% during the outages. I'm not sure what's causing that but I suspect it is a byproduct of the internet connections failing or timing out.

My application uses a lot of different AWS services, including RDS, IOT, and MQTT. The Tomcat logs show failed or timed out connections on all of these. I also cannot connect to the instance via ssh during the outage.

I don't think there's anything wrong with my application because the other instance works fine and has never crashed. I also run a dozen similar EC2/Tomcat instances on another AWS account and those have never had this kind of a problem either.

I suppose I'll try to fix it by replacing the instance at some point, but I'd really like to know why this is happening.

Gary Mclean · Answer

Without looking at any logs, it sounds like memory exhaustion.

If you havent, install cloudwatch agent and monitor memory usage.

Suggest upgrading to xlarge

re:Post Agent · Answer

Based on the symptoms you've described, it appears that your EC2 instance is experiencing a network connectivity issue that occurs periodically. This problem could be related to several factors, but one likely cause is a time synchronization issue.

By default, Windows instances use Coordinated Universal Time (UTC). If the instance's time is set to a different time zone and then restarted, it can cause a time offset. This offset can lead to the instance temporarily losing its IP address, which results in network connectivity issues. The instance usually regains network connectivity eventually, but this process can take several hours.

The symptoms you're experiencing align with this scenario:
1. Loss of network connectivity
2. Inability to access the internet or AWS services
3. SSH connection failures
4. The issue resolves after a reboot

To address this problem, you should ensure that the RealTimeIsUniversal registry key is set on your instance. This key allows the instance to use a time zone other than UTC persistently. Without this key, the instance reverts to UTC after a restart, which can cause the connectivity issues you're experiencing.

The CPU usage spike you're observing during the outages could indeed be a result of the network connections failing or timing out, as your application might be continuously attempting to re-establish these connections.

To resolve this issue:
1. Set the RealTimeIsUniversal registry key to 1 on the affected instance.
2. Ensure that the instance is running the recommended Paravirtual (PV) drivers.
3. Check that the instance's time settings are correct and consistent.

If the problem persists after implementing these changes, you may want to investigate other potential causes such as:
1. Security group or network ACL configurations
2. Instance health (run a status check)
3. Underlying hardware issues (in which case, replacing the instance might indeed be necessary)

Remember to regularly back up your data and consider implementing an Auto Scaling group to automatically replace unhealthy instances if this continues to be an issue.  
**Sources**  
[Troubleshoot issues with Amazon EC2 Windows instances - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/win-ts-common-issues.html)  
[Troubleshoot website connectivity on EC2 Windows instance | AWS re:Post](https://repost.aws/knowledge-center/ec2-windows-website-connection)  
[Instance resiliency - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-recover.html)

ec2 instance loses network connectivity

Add your answer

Relevant content