Skip to content

EC2 NetworkOut drops to 0 for ~26 min while NetworkIn/CPU stay normal (StatusCheckFailed_Instance) - recurring, only fixed by stop/start not reboot

0

On a t3a.small instance in us-west-2 (Ubuntu 22.04, kernel 6.8.0-1063-aws, in-tree ENA driver - already the latest available via apt, no pending kernel update), NetworkOut dropped to exactly 0 bytes/min and stayed at 0 for ~26 minutes, while NetworkIn continued normally (13-30 KB/min - inbound traffic was still arriving) and CPUUtilization stayed flat (~1.7-2%, no load spike). StatusCheckFailed_Instance = 1 for the duration; StatusCheckFailed_System stayed 0 throughout. No AWS-scheduled maintenance event was present on the instance during this window. The instance was unreachable over HTTP/SSH for the duration and only recovered after a full stop/start - a soft reboot did not clear an identical occurrence of this same symptom previously.

This looks like an asymmetric TX-path hang on the ENA adapter (guest can receive but not transmit). Since the driver is already current, I don't have a guest-side fix to try. Has anyone seen this pattern before, and is there a known host-side/Nitro-level cause or mitigation short of repeatedly stopping/starting the instance?

asked a month ago77 views

2 Answers
0

The pattern you're describing—where NetworkOut drops to zero while NetworkIn continues normally, accompanied by a failed instance status check—suggests a guest operating system networking issue rather than a host-level problem, since the system status check remained healthy throughout.

When an instance fails its instance status check but the system status check passes, this typically indicates that the issue originates within the guest OS network stack or configuration, not at the underlying hardware or hypervisor level. The fact that a stop/start resolves the issue (while a reboot does not) is particularly telling, as a stop/start cycle causes the instance to be placed on new underlying hardware, which can clear certain types of stuck states.

For your specific situation with the ENA driver, even though you're running the latest in-tree driver available via apt, there are a few areas to investigate:

First, verify that enhanced networking with the Elastic Network Adapter is properly activated on your instance. Even with the driver installed, misconfigurations can lead to intermittent networking issues.

Second, check your network interface configuration for any hardcoded settings that might conflict with AWS's dynamic environment. Look for hardcoded MAC addresses in udev configuration files or hardcoded IP addresses in your network configuration. These can cause the network stack to become confused or stuck, particularly after certain types of transient events.

Third, review your network interface naming configuration. If you're experiencing naming conflicts or issues with predictable network interface naming, you might consider deactivating it by adding the net.ifnames=0 kernel parameter to your GRUB configuration and rebuilding GRUB.

Finally, ensure that your primary network interface is configured to start at boot and is set up for DHCP rather than static addressing. Cloud-init should be properly installed and functioning to handle dynamic network configuration in the EC2 environment.

While brief, self-resolving host-level conditions can occasionally affect instances, the 26-minute duration and the specific pattern of symptoms you're seeing point more strongly toward a guest OS networking configuration issue that requires investigation within the instance itself.
Sources
Why EC2 Automatic Recovery May Not Trigger During Brief Status Check Failures | AWS re:Post
Troubleshoot network configuration issues in EC2 Linux instance | AWS re:Post

answered a month ago

AWS
SUPPORT ENGINEER

reviewed a month ago

0

I think the discussion around automatic SES Production Access should focus less on one individual account and more on what these examples may be showing about the process itself.

Two public videos have been shared repeatedly: https://www.youtube.com/watch?v=mTW4qJzCf28 https://www.youtube.com/watch?v=Gr9uPlMqo-M

In both cases, Production Access appears to be granted very quickly. We obviously cannot see AWS’s internal checks, so there may be context missing. Still, these examples are difficult to reconcile with the experience of customers who provide detailed information, configure everything correctly, and then wait for manual review. The bigger question is whether this kind of automatic approval can be repeated at scale. If it can, then the risk is not limited to the approved account. Poor sending can affect shared SES IP reputation before AWS takes action. I think AWS should review these videos as examples of a broader pattern and clarify whether the current auto-approval process is behaving as intended.

answered a month ago

EXPERT

reviewed a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.