Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

ENA Driver Failure and EC2 Auto Recovery – How to Identify Root Cause?

0

Hello AWS Community,

We recently faced an incident on one of our production EC2 servers that resulted in downtime.

Timeline & Observations:

  • The server suddenly became unreachable.
  • AWS system status check failed on the instance.
  • Since this was a production server, we immediately triggered our DR (Disaster Recovery) procedure to minimize downtime.
  • As part of troubleshooting, we also restarted the affected server manually.
  • Later, in the AWS Health Dashboard, we noticed a notification:

"EC2 simplified auto recovery success"

  • Looking at system logs, we did not see any CPU/memory surge (verified with SAR), and no OOM killer or application-related errors.
  • Based on kernel logs and behavior, it seems the failure may be related to the ENA (Elastic Network Adapter) driver.

Our Questions:

  1. What typically causes an ENA driver failure on EC2 instances?
  2. How do we distinguish whether this was an AWS infrastructure-level fault or an OS/driver-level issue inside the instance?
  3. Are there specific AWS metrics, logs, or ENA statistics that we can review for the past few days to find the root cause?
  4. What preventive steps should we take to reduce the risk of ENA-related failures in production workloads?
  5. We understand that AWS auto recovery helped in this case, but since this was a production outage, we’d like to perform a proper root cause analysis and strengthen our preventive measures.

Any insights or best practices would be highly appreciated.

Topics
Management & GovernanceCompute
Tags
AWS HealthAmazon EC2Disaster Recovery/Business ContinuityAWS Management ConsoleAmazon CloudWatch Application Monitoring
Language
English

asked a year ago285 views

1 Answer
0

When an EC2 instance experiences a system status check failure followed by automatic recovery, it typically indicates an underlying hardware or software issue on the AWS infrastructure side.

Based on your observations, particularly the "EC2 simplified auto recovery success" notification in the AWS Health Dashboard, this confirms that AWS detected a system-level issue and automatically recovered your instance. Simplified automatic recovery is a feature that responds to underlying hardware or software problems without requiring user intervention.

Regarding ENA driver failures:

  1. Causes of ENA driver failures: The ENA driver can experience issues due to several factors, including:
  • Memory Mapped I/O (MMIO) read timeouts during initialization or reset processes
  • Underlying hardware failures affecting network connectivity
  • Software issues on the physical host
  1. Distinguishing between AWS infrastructure vs OS/driver issues:
  • System status check failures (0/2) generally indicate AWS infrastructure problems, which fall under AWS's responsibility
  • Instance status check failures (1/2) typically point to OS-level issues within your control
  • The "EC2 simplified auto recovery success" notification confirms this was an infrastructure-level issue
  1. Metrics and logs to review:
  • Check the StatusCheckFailed_System CloudWatch metric to confirm when the system status check failed
  • Review the AWS Health Dashboard for detailed information about the recovery event
  • If you suspect ENA driver issues specifically, look for adapter resets or network interface timeouts in your system logs
  1. Preventive measures:
  • While you can't prevent hardware failures on AWS's side, you can build resilience through:
  • Using Elastic Load Balancing to distribute traffic across multiple instances
  • Implementing Auto Scaling to maintain application availability
  • Ensuring your application can gracefully handle instance restarts
  • Regularly reviewing system logs for early warning signs of network adapter issues

For a thorough root cause analysis, you may need to open a support case with AWS Technical Support if the AWS Health Dashboard doesn't provide sufficient details about the specific hardware or software issue that triggered the recovery.
Sources
Linux Ec2 with system check Failed , how to find the root cause. | AWS re:Post
Verify if automatic instance recovery occurred - Amazon Elastic Compute Cloud
Troubleshoot the Elastic Network Adapter Windows driver - Amazon Elastic Compute Cloud
Automatic instance recovery - Amazon Elastic Compute Cloud

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content