EC2 instance randomly fails 1/2 status checks at 4:30 AM with no user activity — SSH and APIs become unresponsive

When your EC2 instance shows "1/2 checks passed" status and becomes unresponsive, this indicates that either the system status check or the instance status check is failing. This is a common issue that can have several causes.

First, let's understand what these status checks mean:

• System Status Check: Verifies AWS infrastructure that the instance runs on
• Instance Status Check: Monitors the software and network configuration of your instance

Here are some troubleshooting steps you can take:

1. Check System Logs: In the EC2 console, select your instance, then go to Actions > Monitor and troubleshoot > Get System Log. Look for error messages that might indicate what's wrong.

2. Stop and Start the Instance (not just reboot): This migrates your instance to new hardware and can resolve underlying issues. This is different from what you're currently doing manually.

3. Monitor Resource Utilization: Use tools like "top" when you can access the instance to check if CPU or memory usage is causing the problem.

4. Check File System and Configuration: The unresponsiveness could be related to file system issues or incorrect configurations.

5. Consider Instance Recovery: While you mentioned automatic recovery isn't working as expected, ensure it's properly configured. EC2 instances can be set up for automatic recovery when they fail system status checks.

6. Try EC2 Serial Console: If SSH is unresponsive, you might be able to access your instance through the EC2 Serial Console (if your instance type supports it).

7. Force Stop Using AWS CLI: If the instance gets stuck in a stopping state, you can use the AWS CLI command with the --force parameter:

aws ec2 stop-instances --instance-ids your-instance-id --force

For a more permanent solution:

• Consider creating a new instance from a known good AMI and carefully reinstall your required software
• Evaluate if your t2.small instance type provides enough resources for your workload
• Implement a monitoring solution that can detect when your instance becomes unresponsive and trigger recovery actions

If the problem persists after trying these steps, it might be worth contacting AWS Support, as there could be an underlying issue with the EC2 infrastructure in your region or availability zone.
Sources
Every day I get “1/2 checks passed” | AWS re:Post
Amazon EC2 Instance from Custom AMI Health Check Failing | AWS re:Post
EC2 instance not restarting | AWS re:Post
EC2 instances status check for system or status check for instance automatically gets failed | AWS re:Post

Hello. You may want to pay particular attention to the instance capacity. Is the instance size/type appropriate for the workloads you're running? Also, is there a scheduled job that possibly gets triggered at that specific time when the instance's status check fails?