How do I resolve the "Kernel panic - not syncing" error in my Amazon EC2 instance?

6 minute read

I want to upgrade my kernel or reboot my Amazon Elastic Compute Cloud (Amazon EC2) Linux instance because of missing initramfs or kernel modules. However, I receive the "Kernel panic - not syncing" error.

Short description

The "Kernel panic - not syncing" error occurs when the device or address doesn't exist. To resolve this issue, launch a temporary EC2 instance, and then attach the faulty root disk as a secondary drive to get the root cause.

Note: To launch a temporary instance, you must stop and start your instance. Before you launch the temporary instance, back up your instance with an Amazon Elastic Block Store (Amazon EBS) snapshot or an Amazon Machine Image (AMI).

Resolution

Note: The following resolution steps are only for Amazon Linux 2 (AL2), Amazon Linux 2023 (AL2023), Fedora 16+, and Red Hat Enterprise Linux (RHEL) 7+.

Configure your instance for a stop and start

Note: When you stop and start an instance, the instance public IP address changes. It's a best practice to use an Elastic IP address to route external traffic to your instance instead of a public IP address. If you use Amazon Route 53, then you might need to update the Route 53 DNS records when the public IP address changes. A stop and start is different from an instance reboot. For more information, see How EC2 instance stop and start works.

Before you stop and start your instance, take the following actions:

If your instance uses an instance store, then save the volume data on the instance store to persistent storage. For example, save the data to Amazon EBS volumes or Amazon Simple Storage Service (Amazon S3) buckets.
Important: Amazon EC2 deletes instance store data when you stop the instance.
Create a snapshot of your EBS volume. If you experience issues, then use the snapshot to restore your instance.
Temporarily remove the instance from its Amazon EC2 Auto Scaling group so that you don't accidentally terminate the instance when you stop it.
Note: Amazon EC2 Auto Scaling might terminate stopped instances in an Auto Scaling group based on your scale-in protection settings. Instances that you launch with Amazon EMR, AWS CloudFormation, or AWS Elastic Beanstalk might be in an Auto Scaling group.
Set the instance shutdown behavior to Stop to make sure that the instance doesn't terminate when you stop it.

Launch a rescue instance to troubleshoot your issue

Complete the following steps:

Get the volume ID and device name for the original instance's root EBS volume.
Stop the original instance.
Launch a temporary instance from an AMI with the same Linux operating system (OS) version in the same Availability Zone.
Detach the root volume from the original instance and attach the volume to the temporary instance as a secondary volume. Note the volume device name.
Use SSH to connect to the temporary instance.
To change to the root user, run the following command:
```
sudo su
```

To get the block device name and partition, run the following command from the temporary instance:

lsblk -f -o +SERIAL

Example output:

NAME          FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS SERIAL  
nvme0n1                                                                                          vol0899d001820438d22  
nvme0n1p1     xfs          /     4148696a-4df2-4559-bf3c-1fb280c80449    6.3G    21% /  
nvme0n1p127  
nvme0n1p128   vfat   FAT16       3303-2E6D                               8.7M    13% /boot/efi  
nvme1n1                                                                                          vol080672f125bafdd25  
nvme1n1p1  
nvme1n1p2     vfat   FAT16 ESP   D0C8-23B6  
€nvme1n1p3    xfs          root  895913b9-13ac-4ee6-934c-1d6224d5d205

Note: The preceding output is from an AWS Nitro instance. Match the SERIAL column to the volume you attached in step 4 and identify the root partition.

If the root file system is on a partition, then mount that partition instead of the whole device nvme1n1. Mounting the device directly fails when the file system is on a partition. For an instance that's built on the AWS Nitro System, the device uses the /dev/nvme[0-26]n1 format. Run the following command to mount the volume on the /mnt directory:
```
mount -o nouuid /dev/nvme1n1p3 /mnt
```
Note: Replace nvme1n1p3 with the root partition from step 7. On Xen instances, use the blkfront name. For example, /dev/xvdf1.
If /boot or /boot/efi is on a separate partition, then run the following command to mount it inside the root file system:
```
mount -o nouuid  /dev/nvme1n1p1 /mnt
```
Note: Replace partition name with the block device name that you identified in step 7. If a dedicated /boot partition also exists, then mount /boot before /boot/efi.
To create a chroot environment in the /mnt directory, run the following command:
```
for i in dev proc sys run; do mount -o bind /$i /mnt/$i; done; chroot /mnt
```
Note: The preceding command bind-mounts the /dev, /proc, /sys, and /run directories from the original root file system. This configuration allows processes that run inside the chroot environment to access the system directories.

To create a backup of the initramfs in the / directory, run the following command:

for file in /boot/initramfs-*.img; do cp "${file}" "/$(basename "$file")_$(date +%Y%m%d)"; done

To list the default kernel, run the following command:

grubby --default-kernel

Example output:

/boot/vmlinuz-5.15.156-102.160.amzn2.x86_64

Note: The preceding output lists the kernels that boot at startup.

To list the kernels and initramfs in the boot directory, run the following command:

ls -lh /boot/vmlinuz* && ls -lh /boot/initr*

Example output:

-rwxr-xr-x. 1 root root 9.7M Apr 23 20:37 /boot/vmlinuz-5.10.215-203.850.amzn2.x86_64
-rwxr-xr-x. 1 root root 9.9M Apr 23 17:00 /boot/vmlinuz-5.15.156-102.160.amzn2.x86_64 
-rw-------. 1 root root 12M May 3 23:45 /boot/initramfs-5.10.215-203.850.amzn2.x86_64.img 
-rw-------. 1 root root 9.8M May 14 08:03 /boot/initramfs-5.15.156-102.160.amzn2.x86_64.img

Note the vmlinuz kernel files that have corresponding initramfs files.

To rebuild the initramfs, run the following command:

dracut --force --verbose /boot/initramfs-kernelVersion.img kernelVersion

Note: Replace kernelVersion with the latest kernel version.

To determine whether the instance boots on UEFI or BIOS, run the following command:

boot_mode=$(ls /sys/firmware/efi/efivars >/dev/null 2>&1 && echo "EFI" || echo "BIOS"); echo "Boot mode detected: $boot_mode"

For instances that boot on BIOS, run the following command to update the grub configuration:
```
grub2-mkconfig -o /boot/grub2/grub.cfg
```
Note: You might receive the "device-mapper: reload ioctl on osprober-linux-xvda2 (253:0) failed: Device or resource busy Command failed" error message when you run the preceding command. To resolve this issue, add the GRUB_DISABLE_OS_PROBER=true parameter to the /etc/default/grub file, and then run the command again.

For instances that boot on UEFI, run the following command based on your OS to update the grub configuration.
Amazon Linux:

grub2-mkconfig -o /boot/efi/EFI/amzn/grub.cfg

Note: AL2 reaches end of life on June 30, 2026. Make sure to migrate AL2 instances to AL2023 before then.
Fedora 16+:

grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

RHEL 7 and RHEL 8:

grub2-mkconfig -o /boot/efi/EFI/amzn/grub.cfg

RHEL 9+:

grub2-mkconfig -o /boot/grub2/grub.cfg

To exit and detach the volume, run the following command:

exit; umount -fl /mnt

Detach the volume from the temporary instance and attach it to the original instance as a root device. Use the device name that you noted in step 4.
Connect to the original instance.