EC2 Ubuntu-20 Instance with error: `Instance reachability check failed` on first boot from our custom AMI with Flask and UWSGI

Question

On checking the logs I have what looks like the issue:
 
```
[   11.362319] cloud-init[404]: Cloud-init v. 21.2-3-g899bfaa9-0ubuntu2~20.04.1 running 'init-local' at Fri, 11 Feb 2022 11:34:54 +0000. Up 11.33 seconds.
[   11.907849] cloud-init[404]: 2022-02-11 11:34:54,987 - util.py[WARNING]: failed stage init-local
[   11.915165] cloud-init[404]: failed run of stage init-local
[   11.916227] cloud-init[404]: ------------------------------------------------------------
[   11.917820] cloud-init[404]: Traceback (most recent call last):
[   11.918709] cloud-init[404]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 652, in status_wrapper
[   11.920231] cloud-init[404]:     ret = functor(name, args)
[   11.921127] cloud-init[404]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 361, in main_init
[   11.922608] cloud-init[404]:     init.apply_network_config(bring_up=bool(mode != sources.DSMODE_LOCAL))
[   11.923969] cloud-init[404]:   File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 744, in apply_network_config
[   11.925987] cloud-init[404]:     return self.distro.apply_network_config(netcfg, bring_up=bring_up)
[   11.927315] cloud-init[404]:   File "/usr/lib/python3/dist-packages/cloudinit/distros/__init__.py", line 210, in apply_network_config
[   11.928971] cloud-init[404]:     dev_names = self._write_network_config(netconfig)
[   11.930276] cloud-init[404]:   File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 116, in _write_network_config
[   11.932021] cloud-init[404]:     return self._supported_write_network_config(netconfig)
[   11.933394] cloud-init[404]:   File "/usr/lib/python3/dist-packages/cloudinit/distros/__init__.py", line 122, in _supported_write_network_config
[   11.935177] cloud-init[404]:     renderer.render_network_config(network_config)
[   11.936240] cloud-init[404]:   File "/usr/lib/python3/dist-packages/cloudinit/net/renderer.py", line 54, in render_network_config
[   11.937853] cloud-init[404]:     return self.render_network_state(
[   11.938761] cloud-init[404]:   File "/usr/lib/python3/dist-packages/cloudinit/net/netplan.py", line 236, in render_network_state
[   11.940368] cloud-init[404]:     self._netplan_generate(run=self._postcmds)
[   11.941368] cloud-init[404]:   File "/usr/lib/python3/dist-packages/cloudinit/net/netplan.py", line 243, in _netplan_generate
[   11.942958] cloud-init[404]:     subp.subp(self.NETPLAN_GENERATE, capture=True)
[   11.944114] cloud-init[404]:   File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 293, in subp
[   11.945542] cloud-init[404]:     raise ProcessExecutionError(stdout=out, stderr=err,
[   11.946643] cloud-init[404]: cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
[   11.948013] cloud-init[404]: Command: ['netplan', 'generate']
[   11.948892] cloud-init[404]: Exit code: 1
[   11.949511] cloud-init[404]: Reason: -
[   11.950124] cloud-init[404]: Stdout:
[   11.950738] cloud-init[404]: Stderr: Traceback (most recent call last):
[   11.951695] cloud-init[404]:           File "/usr/sbin/netplan", line 20, in 
[   11.952808] cloud-init[404]:             from netplan import Netplan
[   11.953782] cloud-init[404]:           File "/usr/share/netplan/netplan/__init__.py", line 18, in 
[   11.955123] cloud-init[404]:             from netplan.cli.core import Netplan
[   11.956206] cloud-init[404]:           File "/usr/share/netplan/netplan/cli/core.py", line 24, in 
[   11.957557] cloud-init[404]:             import netplan.cli.utils as utils
[   11.958552] cloud-init[404]:           File "/usr/share/netplan/netplan/cli/utils.py", line 26, in 
[   11.959914] cloud-init[404]:             import netifaces
[   11.960736] cloud-init[404]:         ModuleNotFoundError: No module named 'netifaces'
[   11.961989] cloud-init[404]: ------------------------------------------------------------
[[0;1;31mFAILED[0m] Failed to start [0;1;39mInitial cloud-init job (pre-networking)[0m.
See 'systemctl status cloud-init-local.service' for details.
```
A simple reboot corrects the problem but as I need to launch this via a Launch Config for an autoscaling group I cannot reboot to get this working.

Anyone have any ideas?

Answer

Hi Dub ,

I understand that your instance is getting an ‘instance reachability check failed’ error.Please correct me if I misunderstood.

The output you provided indicates that there is some issues with the cloud-init service as it has failed to run during the instance’s boot time.The solution is removing the cloud-init package that’s currently present and re-install cloud-init package again so that it is installed correctly.

You won’t be able to access the instance because of the failing status checks so you need to create a recovery instance.Firstly you will need to detach the root volume of the instance and attach it to the recovery instance, by following this link[1] and mount the volume to /mnt directory.Once it is mounted ,do the following :

```
# mount bind the pseudo filesystems like /proc , /sys and /dev  and chroot to /mnt
$ sudo mount -o bind /proc /mnt/proc 
$ sudo mount -o bind /dev /mnt/dev
$ sudo mount -o bind /sys /mnt/sys
$ sudo chroot /mnt
```
Next up you need to do a simple ping to check if the instance is reachable to the internet.Once that is done you must remove the current cloud-init package followed by re-installing a new cloud-init package following the commands below

```
# apt update
# apt remove cloud-init
# apt purge cloud-init -y
# apt install cloud-init -y
# rm -rf /var/lib/cloud/*      — remove existing configurations corresponding to the cloud-init service
```
When you are done ,you need to exit the chroot then unmount the volume that you mounted at the beginning.You should then re-attach the volume from the recovery instance to the first instance.

You will then need to take an image from the fixed instance,Please follow link[2] and use the working AMI for your instances.

I hope this was helpful!

Reference

[1]https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-attaching-volume.html

[2]https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/tkv-create-ami-from-instance.html

EC2 Ubuntu-20 Instance with error: `Instance reachability check failed` on first boot from our custom AMI with Flask and UWSGI

Relevant content