Instance storage not showing up on ec2 instance (Ubuntu 5.19)

Question

I boot an i3.large instance daily which is supposed to have 475 GB NVMe SSD instance storage. Recently on two occasions the disk did not show up and did not appear in lsblk. Rebooting did not fix it and it was necessary to create a new instance from scratch to access the instance storage. Is there a way to debug or fix this problem permanently?

Accepted Answer

This seems to be a problem with the Amazon firmware described here (https://lore.kernel.org/all/3E46620F-C371-4108-9A74-AA1A624AA9A5@amazon.de/T/)

The problem can be confirmed with `journalctl -b | grep nvme` which will show `nvme nvme0: IO queues not created`. Under correct operation it should show `nvme nvme0: 15/0/0 default/read/poll queues`.

The problem is triggered by 5.19 kernel and it seems that rolling back the kernel offers a workaround.

#### Kernel downgrade instructions

Edit `/etc/default/grub` to say `GRUB_DEFAULT=saved`.
Then run `sudo update-grub` to regenerate the configuration.

`grep menuentry /boot/grub/grub.cfg` to find the menu ids of the old kernel.

In my case `gnulinux-advanced-cda0c0a7-e64a-4413-85a5-a0235f6f567f>gnulinux-5.15.0-1031-aws-advanced-cda0c0a7-e64a-4413-85a5-a0235f6f567f`.

Run `sudo grub-set-default "gnulinux-advanced-cda0c0a7-e64a-4413-85a5-a0235f6f567f>gnulinux-5.15.0-1031-aws-advanced-cda0c0a7-e64a-4413-85a5-a0235f6f567f"`

Now reboot and it should use the previous kernel.

Confirm the correct kernel has booted by running `uname -a`. `5.15.0-1031-aws` should be visible.

Instance storage not showing up on ec2 instance (Ubuntu 5.19)

Kernel downgrade instructions

Relevant content