Boot timeout starting instance as t3.small type, works fine as t2.small

Question

Hi,  
my Centos7 AMI start fine as t2.small but does't boot using instance type t3.small or any Nitro Based instance type, with system log:

```
...
many dracut-initqueue timeout
...
[  184.749683] dracut-initqueue[231]: Warning: dracut-initqueue timeout - starting timeout scripts
[  185.258984] dracut-initqueue[231]: Warning: dracut-initqueue timeout - starting timeout scripts
[  185.262729] dracut-initqueue[231]: Warning: Could not boot.
[  185.265338] dracut-initqueue[231]: Warning: /dev/centos/root does not exist
[  185.268902] dracut-initqueue[231]: Warning: /dev/centos/swap does not exist
[  185.272733] dracut-initqueue[231]: Warning: /dev/mapper/centos-root does not exist
         Starting Setup Virtual Console...
[  OK  ] Started Setup Virtual Console.
         Starting Dracut Emergency Shell...
Warning: /dev/centos/root does not exist
Warning: /dev/centos/swap does not exist
Warning: /dev/mapper/centos-root does not exist
Generating "/run/initramfs/rdsosreport.txt"
Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.
```

I double checked all test (nitro support: ENA, NVMe, ecc.) and run nitro scripts as explained:

I tried many Fix, also editing fstab and /boot/grub2/grub,cfg using UUID instead of fixed device name, but nothings worked... I cannot find what stop on the boot process !  
  
Edited by: Roxyrob on Jul 10, 2020 9:22 AM  
  
Edited by: Roxyrob on Jul 10, 2020 9:26 AM  
  
Edited by: Roxyrob on Jul 10, 2020 9:34 AM

Answer

Found solution. I think aws documentation can be improved (check script and documentation for new nitro systems). Script erroneously return  "OK" for NVMe check. In fact nvme driver (e.g. nvme.ko.xz) was present in the OS (modeprobe nvme), but not in initramfs, allowing boot unable to discover and use new nitro based nvme devices.  
  
To add nvme driver in initramfs (available at device discover and mount phase):

```
echo 'add_drivers+=" nvme "' > /etc/dracut.conf.d/nvme.conf
dracut -f -v
lsinitrd /boot/initramfs-$(uname -r).img | grep nvme.ko
```

other configuration make instance more robust:  
  
For (ENA):  
  
(optional - network availability in initramfs phase)

```
echo 'add_drivers+=" ena "' > /etc/dracut.conf.d/ena.conf
dracut -f -v
lsinitrd /boot/initramfs-$(uname -r).img | grep ena.ko
```

(optional - avoid network device name change between reboot)

```
add net.ifnames=0 to kernel options (e.g in GRUB_CMDLINE_LINUX inside /etc/default/grub and executing grub2-mkconfig -o /boot/grub2/grub.cfg. OS distribution and version dependent)
```

(optional - avoid inconsistent disk name)

```
use UUID instead of device names in /etc/fstab and boot loader (e.g. for GRUB in /boot/grub2/grup.cfg root=UUID=...).
```

Edited by: Roxyrob on Jul 10, 2020 12:17 PM  
  
Edited by: Roxyrob on Jul 10, 2020 12:31 PM

Boot timeout starting instance as t3.small type, works fine as t2.small

Relevant content