Redhat 8 instance doesn't boot

Question

I have a Redhat 8 instance that will not boot. After startup I get an Instance Status Check failure:  Instance reachability check failed. The serial console is blank and I can not reach the instance via ssh. I have attached the volume to a working instance and can mount /, /boot and /boot/efi partitions. The data looks looks fine. I have run xfs_repair on the XFS filesystems and fsck on the FAT.

sudo mount /dev/nvme1n1p4 /mnt
    sudo mount /dev/nvme1n1p3 /mnt/boot
    sudo mount /dev/nvme1n1p2 /mnt/boot/efi
    sudo chroot /mnt

After mounting the system there are no new logs since the last working shutdown. Should I try and save this volume or since I can get the volume mounted I just copy my data to a new instance?

I've run AWSSupport-ExecuteEC2Rescue and get

----------[Diagnostic Results]----------

module run/openssh               
module run/enadiag               [FAILURE] ENA problems found on ens5.
module run/arpcache              [SUCCESS] Aggressive arp caching is disabled.
module run/arpignore             [SUCCESS] arp ignore is disabled for all interfaces.
module run/asymmetricroute       [SUCCESS] No duplicate subnets found.
module run/conntrackfull         [SUCCESS] No conntrack table full errors found.
module run/consoleoverload       [SUCCESS] No serial console overload found
module run/duplicatefslabels     [SUCCESS] No duplicate filesystem labels found.
module run/fstabfailures         [SUCCESS] /etc/fstab rewritten
module run/hungtasks             [SUCCESS] No hung tasks found
module 
---Output truncated---

Not sure what to do with the ENA failure if anything.

re:Post Agent · Answer

Based on the information provided, it appears your Red Hat 8 EC2 instance is experiencing a boot failure. Since you can successfully mount the volume and access the data, this suggests the issue is likely related to the boot process rather than data corruption.

The ENA (Elastic Network Adapter) failure detected by the AWSSupport-ExecuteEC2Rescue tool could be contributing to the boot problem, as network issues can sometimes prevent an instance from fully booting or becoming reachable.

Given that you've already taken several troubleshooting steps without success, you have two main options:

1. Attempt further recovery:
   - Try updating the ENA driver on the mounted volume.
   - Check and update the GRUB bootloader configuration.
   - Verify that the kernel and initramfs are present and not corrupted.
   - Examine any custom startup scripts or services that might be interfering with the boot process.

2. Create a new instance and migrate data:
   - Since you can access the data by mounting the volume on another instance, this might be the quickest and most reliable solution.
   - Launch a new Red Hat 8 instance.
   - Attach the old volume to the new instance.
   - Copy your essential data and configurations to the new instance.
   - Reconfigure any necessary services on the new instance.

Given that you've already performed several recovery attempts without success, and considering the critical nature of the server, option 2 might be the most pragmatic approach. It allows you to get back up and running quickly while minimizing the risk of ongoing issues.

After migrating to a new instance, you can continue investigating the root cause of the boot failure on the original volume without impacting your operations. This approach also provides an opportunity to review and potentially improve your instance configuration and backup strategies to prevent similar issues in the future.  
**Sources**  
[Boot fails in Amazon EC2 Linux instance | AWS re:Post](https://repost.aws/knowledge-center/ec2-linux-boot-fails)  
[One of my ec2 instance (Amazon Linux) stopped booting suddenly today! | AWS re:Post](https://repost.aws/questions/QUCJpoBsK6QbC3AhcjDKq9qw/one-of-my-ec2-instance-amazon-linux-stopped-booting-suddenly-today)  
[Assistance Required for Non-Responsive EC2 Instance | AWS re:Post](https://repost.aws/questions/QUjKFD51VuTOaGivk4KEor-w/assistance-required-for-non-responsive-ec2-instance)  
[Can't connect to Linux instance even after the health status checks pass | AWS re:Post](https://repost.aws/knowledge-center/connect-linux-health-check-pass)

Redhat 8 instance doesn't boot

Add your answer

Relevant content