EC2 Amazon Linux Stuck in Grub boot loader after reboot

The issue you're experiencing with your Amazon Linux instance being stuck in the GRUB boot loader after a reboot is a common problem that can occur due to various reasons, including updates or configuration changes. Your approach to troubleshooting is on the right track, but there are a few adjustments we can make to resolve the issue.

First, it's important to note that when reinstalling GRUB, you should target the entire disk device, not just a partition. In your case, you're trying to install GRUB on /dev/nvme1n1p1, which is a partition, not the entire disk. This is likely why you're seeing the error message about not having a corresponding BIOS drive.

To fix this, you should run the grub-install command on the entire disk device. In your case, that would be /dev/nvme1n1. Here's what you should do:

1. While still in the chroot environment, run: grub-install /dev/nvme1n1

2. After successfully installing GRUB, update the GRUB configuration: update-grub

3. Exit the chroot environment: exit

4. Unmount all the bound directories and the main partition: for m in sys proc dev run; do umount /mnt/$m; done umount /mnt

5. Detach the volume from your rescue instance and reattach it to your original instance.

6. Start your original instance.

If the issue persists, you might need to check the GRUB configuration file (/boot/grub/grub.cfg or /boot/grub/menu.lst) to ensure it's correctly configured. Look for any recent changes or updates that might have affected the GRUB configuration.

Additionally, make sure that your instance's boot volume is correctly attached and that you're using the appropriate GRUB image for your system (first disk or first partition on the first disk).

If you continue to face issues, you might consider creating a new AMI with a properly configured GRUB at the standard location (/boot/grub/menu.lst) and launching a new instance from this AMI.

Remember to always take snapshots of your volumes before making significant changes to ensure you can recover your data if needed.
Sources
EC2 Instance not booting, stuck on grub rescue screen | AWS re:Post
Troubleshoot your Amazon EC2 instance using the EC2 Serial Console - Amazon Elastic Compute Cloud
Troubleshoot Amazon EC2 Linux instances with failed status checks - Amazon Elastic Compute Cloud

It does not seem to be working with only nvme1n1

mount /dev/nvme1n1 /mnt
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/nvme1n1, missing codepage or helper program, or other error.

It does work fine with mount /dev/nvme1n1p1

so no progress yet...

Let me provide a detailed solution based on AWS documentation and best practices:

1. First, your approach to mounting the volume is correct.

Here's the correct sequence of commands based on AWS documentation:

# Mount the partition
mount /dev/nvme1n1p1 /mnt

# Mount necessary system directories
for i in /dev /proc /sys /run; do mount --bind $i /mnt$i; done

# Enter chroot environment
chroot /mnt

2. For GRUB installation, since you're using Amazon Linux with GPT partitioning, you need to ensure both the BIOS boot partition and the main partition are properly handled. Kindly try these steps:

# Install grub2 to the device (not partition)
grub2-install --target=i386-pc --recheck /dev/nvme1n1

# Generate grub configuration
grub2-mkconfig -o /boot/grub2/grub.cfg

3. If you're still experiencing issues, verify the BIOS boot partition (nvme1n1p128) is properly flagged. You can check this using:

parted /dev/nvme1n1 print

The BIOS boot partition should have the "bios_grub" flag set.

4. Additional verification steps from AWS documentation:

# Verify boot configuration
ls -l /boot/grub2
ls -l /boot/vmlinuz*
ls -l /boot/initramfs*

5. Before detaching the volume:

# Exit chroot
exit

# Unmount everything
for i in /dev /proc /sys /run; do umount /mnt$i; done
umount /mnt

Sources:

• AWS EBS Volume Documentation
• AWS EC2 Troubleshooting Guide
• AWS Knowledge Center: How do I troubleshoot boot issues on my Amazon EC2 instance?

If these steps don't resolve the issue, you might want to consider:

1. Checking system logs in /var/log/ for any boot-related errors
2. Verifying that the kernel and initramfs files referenced in the GRUB configuration actually exist
3. Creating an AMI from the volume after repairs for backup purposes