EC2 Instance Failing Status Checks with No Logs Update

Question

Hello AWS Community,

I'm facing a critical issue where my EC2 instance is failing status checks since yesterday. This problem is recurring today as well. Surprisingly, the system logs on the instance do not seem to update correspondingly, leaving me with no clear indication of what might be causing these failures.

Here's what I've observed:

- The AWS Management Console shows that the instance has failed status checks.
- The issue has persisted across two consecutive days.
- System logs within the instance do not show new entries around the time of the status check failures.
- There are no obvious signs of resource exhaustion or operational overload.

I'm looking for guidance on the following:

- How can I diagnose the cause of these status check failures more effectively?
- Is there a way to access more detailed diagnostic information from the AWS side that could shed light on these incidents?
- Are there recommended steps to follow when the instance logs do not update during such failures?
- Any advice to troubleshoot this issue and prevent future occurrences would be highly appreciated.

Thank you for your help.

Answer

When you mention there's nothing in the logs around the time that the instance check fails, does this mean you can still login to the EC2 instance and look at the logs while its status is failed? Or are you having to restart the instance and then look back at the logs from the time of the failure?

Troubleshooting suggestions are here although it sounds like you have gone through a lot of them before posting your question https://repost.aws/knowledge-center/ec2-linux-status-check-failure

And here https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstances.html

I note what you say about resource exhaustion not appearing to be a factor, it would still be worth putting CloudWatch Agent on the host to get a more detailed view of resource consumption https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html

What is the instance type, and what sort of workload is it running?

EC2 Instance Failing Status Checks with No Logs Update

Relevant content