Skip to content

EC2 instance rebooted unexpectedly on 2026-07-26 — StatusCheckFailed_System, suspected underlying-host failure

0

An EC2 instance (t3.xlarge, us-east-1, Availability Zone us-east-1c) rebooted unexpectedly on July 26, 2026, at approximately 20:31 UTC. I am trying to determine whether the evidence indicates an underlying AWS infrastructure or host issue.

Timeline — UTC

  • 20:05:55 — SSM Agent reported errors reaching the Instance Metadata Service at 169.254.169.254.
  • 20:06:38 — Last OS log entry. The system appeared normal, with no shutdown sequence.
  • 20:10–20:28 — StatusCheckFailed_System = 1 continuously. StatusCheckFailed_Instance = 1 was also reported during the same period.
  • 20:31:23 — The instance booted with a fresh kernel start.

Evidence that the reboot was not guest-initiated

  • The OS journal contains no orderly systemd shutdown or reboot sequence. Logging ends abruptly at 20:06:38, followed by a fresh kernel boot at 20:31:23.
  • No user-initiated reboot or shutdown command was identified.
  • No scheduled OS shutdown was configured.
  • No OS or kernel update occurred around this time, and the kernel version was unchanged before and after the reboot.
  • No kernel panic, OOM event, hung-task warning, MCE, or guest-visible hardware error was found.
  • CloudTrail contains no RebootInstances, StopInstances, or StartInstances API calls for the relevant period.
  • The EC2 LaunchTime did not change, which appears consistent with a reboot or reset rather than a stop/start operation.

Questions

  1. Does the sustained system status-check failure, together with the abrupt loss of guest logging and absence of a shutdown sequence, strongly indicate an underlying-host or AWS infrastructure issue?
  2. Since the instance is EBS-backed, would a controlled stop/start be recommended to migrate it to another host and reduce the risk of recurrence?
  3. Without access to account-specific AWS Health API information, are there other ways to determine whether a host recovery, retirement, maintenance, or infrastructure event occurred?
  4. Is there any additional metric or console information that would help distinguish automatic EC2 recovery from another type of infrastructure reset?

The instance is currently running normally, and all status checks are passing.

asked 17 days ago88 views

3 Answers
0

Based on the evidence you've described, the sustained StatusCheckFailed_System metric combined with the abrupt cessation of guest logging and absence of any shutdown sequence does strongly suggest an underlying hardware or software issue on the AWS infrastructure side. When system status checks fail, this typically indicates problems with the underlying host that are outside the guest operating system's control—such as loss of network connectivity to the physical host, hardware degradation, or software issues on the underlying system.

Regarding whether automatic instance recovery occurred: You can verify this by checking your Health Dashboard for specific recovery events. Look for events such as AWS_EC2_SIMPLIFIED_AUTO_RECOVERY_SUCCESS or AWS_EC2_SIMPLIFIED_AUTO_RECOVERY_FAILURE for simplified automatic recovery, or AWS_EC2_INSTANCE_AUTO_RECOVERY_SUCCESS and AWS_EC2_INSTANCE_AUTO_RECOVERY_FAILURE for CloudWatch action-based recovery. These events would confirm whether AWS automatically attempted to recover your instance in response to the detected issue. The StatusCheckFailed_System metric showing a value of 1 during that window confirms that a system status check failure was detected, which is a common trigger for automatic recovery.

For an EBS-backed instance like yours, performing a controlled stop and start would indeed migrate the instance to new underlying hardware. This can be a prudent step to reduce the risk of recurrence if you suspect the previous host experienced issues. A stop/start operation changes the underlying host while preserving your instance configuration and EBS volumes.

Without access to account-specific information, the Health Dashboard remains your primary resource for identifying host recovery, retirement, maintenance, or infrastructure events associated with your instances. The dashboard provides visibility into AWS service events that may have affected your resources. Additionally, continuing to monitor the StatusCheckFailed_System CloudWatch metric can help you detect if similar issues recur—a value of 0 indicates the system status check passed, while 1 indicates a failure.

To distinguish automatic recovery from other infrastructure events, the Health Dashboard events are your most reliable indicator. The presence of specific recovery event codes would confirm that automatic recovery was triggered. The fact that your LaunchTime didn't change is consistent with a reboot or recovery action rather than a full stop/start cycle, which would update that timestamp.
Sources
Verify if automatic instance recovery occurred - Amazon Elastic Compute Cloud
Understand why EC2 instance stopped | AWS re:Post

answered 17 days ago

EXPERT

reviewed 17 days ago

0

Hello.

20:10–20:28 — StatusCheckFailed_System = 1 continuously. StatusCheckFailed_Instance = 1 was also reported during the same period.

If "StatusCheckFailed_System" was recorded as 1 in the CloudWatch metrics during that time period, there is a high probability that an issue occurred on the physical AWS host where the EC2 instance was running.
An explanation is also provided in the following document.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html

System status checks monitor the AWS systems on which your instance runs. These checks detect underlying problems with your instance that require AWS involvement to repair. When a system status check fails, you can choose to wait for AWS to fix the issue, or you can resolve it yourself. For instances backed by Amazon EBS, you can stop and start the instance yourself, which in most cases results in the instance being migrated to a new host. For instances backed by instance store (supported only for Linux instances), you can terminate and replace the instance. Note that instance store volumes are ephemeral and all data is lost when the instance is stopped.

  • Loss of network connectivity
  • Loss of system power
  • Software issues on the physical host
  • Hardware issues on the physical host that impact network reachability
EXPERT

answered 17 days ago

EXPERT

reviewed 17 days ago

0

Hello @Gennadiy,

Details as per your request.

  1. Does the sustained system status-check failure, together with the abrupt loss of guest logging and absence of a shutdown sequence, strongly indicate an underlying-host or AWS infrastructure issue? Answer: Yes, the evidence strongly suggests an underlying host or AWS infrastructure issue. The key points are:

System Status Check Failure: The StatusCheckFailed_System status indicates that AWS detected issues with the underlying hardware or infrastructure. Abrupt Loss of Logging: The OS logs end abruptly without any shutdown sequence, which rules out a graceful shutdown or reboot initiated by the guest OS. No User-Initiated Actions: There were no user-initiated reboot or shutdown commands, and no API calls (RebootInstances, StopInstances, StartInstances) were recorded in CloudTrail. No OS Updates or Kernel Panics: There were no OS or kernel updates, and no indications of kernel panics or other guest-visible hardware errors. These factors collectively point towards an issue with the underlying host or AWS infrastructure.

  1. Since the instance is EBS-backed, would a controlled stop/start be recommended to migrate it to another host and reduce the risk of recurrence? Answer: Yes, performing a controlled stop/start is a recommended practice. This will detach the instance from the current host and attach it to a different one, which can help mitigate the risk of recurrence if the issue was indeed host-specific. Here are the steps:

Stop the Instance: Use the AWS Management Console, CLI, or SDKs to stop the instance. Start the Instance: After the instance is stopped, start it again. AWS will place it on a different host. 3. Without access to account-specific AWS Health API information, are there other ways to determine whether a host recovery, retirement, maintenance, or infrastructure event occurred? Answer: While direct access to AWS Health API would provide the most specific information, there are other ways to gather insights:

AWS Personal Health Dashboard: Check the Personal Health Dashboard for any events that might have affected your region or Availability Zone. AWS Service Health Dashboard: Though more general, this can provide insights into any widespread issues. CloudTrail Events: Review CloudTrail for any relevant events around the time of the incident. AWS Support: If you have a support plan

answered 15 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.