EC2 instance fails to start after changing instance type from t2 to t3

Question

I tried to update an EC2 instance from t2 to t3. Since the AZ I was running the instance in did not support t3 instances. I stopped the instance, created an image, and then tried to create an instance from that image in us-east-1c.

The instance is running RockyOS v. 8.5.

The instance did not start.

Using the serial console, it appears as though the EBS volume was not detected.

I verified that the ENA and NMVE drivers were installed.

I tried a series of experiments, where I created new instances from the original AMI and I was able to create t2 instances, stopped them, created images, and then created new t3 instances without issue.

The main difference of course is that the production instance has a lot more data on it, has been updated via dnf update, etc.

I suppose I could just create a brand new t3 instance and migrate the data over, but I would like to understand why I wasn't able to convert the instance from t2 to t3.

Some more information:

The reason that the experiments worked, was because the original AMI was based on RockyOS v. 8.4. This version allows me to migrate between t2 and t3 versions without any issues.

The production instance was updated at some point to version 8.5 and for some reason this version does not boot on t3 (nitro) instance types.

I repeated my experiment, and launched the original AMI in a t2, did an upgrade, then after changing the instance type to t3, the instance does not boot.

While this doesn't provide a solution to the problem, at least not it is reproducible.

So, what is it about Rocky OS v. 8.5 that prevents the migration to a nitro instance?

The modinfo ena and modinfo nvme both show the drivers are present

Answer

I was able to resolve my issue by explicitly added the ena and nvme drivers to the dracut configuration and re-running dracut:

`echo 'add_drivers+=" ena "' >> /etc/dracut.conf.d/ena.conf`

`echo 'add_drivers+=" nvme "' >> /etc/dracut.conf.d/ena.conf`

`dracut -f -vvvv`

This was not needed for Rocky OS v. 8.4, but is needed for v. 8.5.

Answer

It didn't even occurred to me that one should be able to do that, I generally consider instances to be disposable. Anyway, this doesn't answer your question.

[This](https://serverfault.com/questions/869710/is-it-possible-to-change-the-ebs-volume-to-different-availability-zones) seems to suggest you can't use an EBS in us-east-1a for an instance in us-east-1c (eg, the volume needs to be in the same az as the instance). They also suggest that you can make a snapshot across regions (eg, snapshot your existing volume in the zone you want to "move" your instance). This could help with the "volume was not detected" problem. (you will be starting the new instance with the snapshot, not the original volume).

For future proofing, I would (resources permitting) rework the system and split the instance in 2 parts: one volume for the software that should be disposable (you should have an automated way to deploy the software and configuration) and one separate volume for the data (and this is the one you backup and snapshot and babysit :) )

EC2 instance fails to start after changing instance type from t2 to t3

Relevant content