New ec2 instance with GPU using custom AMI takes long time to start up for the first time

Question

I set up a custom AMI for a EC2 G4dn instance which host a GPU web app. On a typical instance startup, it takes about 1 min for the web app to load data from disk and external websites into memory and be ready to serve. However, this startup process takes >10 mins on each new instance I created on this custom AMI, and then the next time the same instance would take the normal 1 min for startup.

So, what is happening differently on each new instance I started for the 1st time, compared to their following startups? How can I prevent this? I am worried that this may affect how autoscaling groups for this web app.

Answer

Without knowing more about the base AMI you're building from this is just part educated-guess / part stab-in-the-dark, but on the first boot could it be doing an SELinux relabel, and has to walk the whole of the directory tree (which takes time)?

Subsequent reboots won't need this, so they are quicker.

Answer

When you create an AMI, a snapshot is created and **stored in S3**. 
Start an instance from the AMI, it takes time to read from S3 when initialize EBS, and if there is a large amount of custom data in the custom AMI, it takes a certain amount of time to initialize.

To reduce this time, you can refer to the following solutions：
https://docs.aws.amazon.com/ebs/latest/userguide/ebs-fast-snapshot-restore.html

But sorry, I don't know how to use this feature with ASG, please let me know if you know, thanks.

New ec2 instance with GPU using custom AMI takes long time to start up for the first time

Relevant content