EC2 Instance reachability check failed after yum upgrade

Question

Instance id: i-096ae05732c55de39  
AMI ID: amzn2-ami-hvm-2.0.20190228-x86_64-gp2 (ami-0de7daa7385332688)  
  
I tried to reboot, stop and start, detach and attach the volume but nothing works. I tried creating a new instance and attach the problematic instance volume to the new instance and I'm getting the same issue with the same logs (see below).  
  
I really hope you can help me log back to the instance or recover the current code and crontab.  
  
Thank you,  
Ido  
  
System logs:  
\[    6.515568] cloud-init\[2107]: Cloud-init v. 19.3-2.amzn2 running 'init' at Sun, 12 Jan 2020 14:50:46 +0000. Up 6.49 seconds.  
  
\[    6.531320] cloud-init\[2107]: ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++  
  
\[    6.531553] cloud-init\[2107]: ci-info: _--------_-------_-----------_-----------_-------_-------------------+  
  
\[    6.531687] cloud-init\[2107]: ci-info: | Device |   Up  |  Address  |    Mask   | Scope |     Hw-Address    |  
  
\[    6.531814] cloud-init\[2107]: ci-info: _--------_-------_-----------_-----------_-------_-------------------+  
  
\[    6.531938] cloud-init\[2107]: ci-info: |  eth0  | False |     .     |     .     |   .   | 06:0a:56:c4:72:d4 |  
  
\[    6.532090] cloud-init\[2107]: ci-info: |   lo   |  True | 127.0.0.1 | 255.0.0.0 |  host |         .         |  
  
\[    6.532209] cloud-init\[2107]: ci-info: |   lo   |  True |  ::1/128  |     .     |  host |         .         |  
  
\[    6.532326] cloud-init\[2107]: ci-info: _--------_-------_-----------_-----------_-------_-------------------+  
  
\[    6.532442] cloud-init\[2107]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++  
  
\[    6.532565] cloud-init\[2107]: ci-info: _-------_-------------_---------_-----------_-------_  
  
\[    6.532683] cloud-init\[2107]: ci-info: | Route | Destination | Gateway | Interface | Flags |  
  
\[    6.532797] cloud-init\[2107]: ci-info: _-------_-------------_---------_-----------_-------_  
  
\[    6.532912] cloud-init\[2107]: ci-info: _-------_-------------_---------_-----------_-------_  
  
\[    6.643586] cloud-init\[2107]: Jan 12 14:50:46 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed \[0/1s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[    7.646859] cloud-init\[2107]: Jan 12 14:50:47 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[    7.666755] cloud-init\[2107]: Jan 12 14:50:47 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[0/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[    8.670012] cloud-init\[2107]: Jan 12 14:50:48 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[    8.690271] cloud-init\[2107]: Jan 12 14:50:48 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[1/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[    9.693515] cloud-init\[2107]: Jan 12 14:50:49 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[    9.713765] cloud-init\[2107]: Jan 12 14:50:50 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[2/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   10.716931] cloud-init\[2107]: Jan 12 14:50:51 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   10.737123] cloud-init\[2107]: Jan 12 14:50:51 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[3/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   11.740529] cloud-init\[2107]: Jan 12 14:50:52 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   11.760948] cloud-init\[2107]: Jan 12 14:50:52 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[4/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   12.764088] cloud-init\[2107]: Jan 12 14:50:53 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   12.785413] cloud-init\[2107]: Jan 12 14:50:53 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[5/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   14.789780] cloud-init\[2107]: Jan 12 14:50:55 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   14.809967] cloud-init\[2107]: Jan 12 14:50:55 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[7/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   16.814238] cloud-init\[2107]: Jan 12 14:50:57 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   16.834230] cloud-init\[2107]: Jan 12 14:50:57 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[9/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   18.838605] cloud-init\[2107]: Jan 12 14:50:59 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   18.859101] cloud-init\[2107]: Jan 12 14:50:59 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[11/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   20.863487] cloud-init\[2107]: Jan 12 14:51:01 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   20.890368] cloud-init\[2107]: Jan 12 14:51:01 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[13/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   22.894642] cloud-init\[2107]: Jan 12 14:51:03 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   22.916020] cloud-init\[2107]: Jan 12 14:51:03 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[15/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   25.921482] cloud-init\[2107]: Jan 12 14:51:06 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   25.942292] cloud-init\[2107]: Jan 12 14:51:06 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[18/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   28.947641] cloud-init\[2107]: Jan 12 14:51:09 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   28.980291] cloud-init\[2107]: Jan 12 14:51:09 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[21/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   31.985667] cloud-init\[2107]: Jan 12 14:51:12 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   32.006417] cloud-init\[2107]: Jan 12 14:51:12 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[24/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   35.011882] cloud-init\[2107]: Jan 12 14:51:15 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   35.032071] cloud-init\[2107]: Jan 12 14:51:15 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[27/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   38.037589] cloud-init\[2107]: Jan 12 14:51:18 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   38.058474] cloud-init\[2107]: Jan 12 14:51:18 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[30/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   42.064758] cloud-init\[2107]: Jan 12 14:51:22 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   42.084761] cloud-init\[2107]: Jan 12 14:51:22 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[34/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   46.091063] cloud-init\[2107]: Jan 12 14:51:26 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   46.111536] cloud-init\[2107]: Jan 12 14:51:26 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[38/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   50.117923] cloud-init\[2107]: Jan 12 14:51:30 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   50.139248] cloud-init\[2107]: Jan 12 14:51:30 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[42/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   54.145806] cloud-init\[2107]: Jan 12 14:51:34 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   54.165882] cloud-init\[2107]: Jan 12 14:51:34 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[46/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   58.172140] cloud-init\[2107]: Jan 12 14:51:38 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   58.191955] cloud-init\[2107]: Jan 12 14:51:38 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[50/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   63.199313] cloud-init\[2107]: Jan 12 14:51:43 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   63.220255] cloud-init\[2107]: Jan 12 14:51:43 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[55/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   68.227799] cloud-init\[2107]: Jan 12 14:51:48 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   68.248621] cloud-init\[2107]: Jan 12 14:51:48 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[60/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   73.255912] cloud-init\[2107]: Jan 12 14:51:53 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   73.275464] cloud-init\[2107]: Jan 12 14:51:53 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[65/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   78.282771] cloud-init\[2107]: Jan 12 14:51:58 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   78.303398] cloud-init\[2107]: Jan 12 14:51:58 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[70/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   83.310696] cloud-init\[2107]: Jan 12 14:52:03 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   83.341005] cloud-init\[2107]: Jan 12 14:52:03 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[75/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   89.349505] cloud-init\[2107]: Jan 12 14:52:09 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   89.370090] cloud-init\[2107]: Jan 12 14:52:09 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[81/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[   95.378513] cloud-init\[2107]: Jan 12 14:52:15 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[   95.399052] cloud-init\[2107]: Jan 12 14:52:15 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[87/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[  101.407361] cloud-init\[2107]: Jan 12 14:52:21 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[  101.429123] cloud-init\[2107]: Jan 12 14:52:21 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[93/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[  107.437417] cloud-init\[2107]: Jan 12 14:52:27 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[  107.460035] cloud-init\[2107]: Jan 12 14:52:27 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[99/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[  113.468430] cloud-init\[2107]: Jan 12 14:52:33 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[  113.488680] cloud-init\[2107]: Jan 12 14:52:33 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[105/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[  120.498447] cloud-init\[2107]: Jan 12 14:52:40 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[  120.519135] cloud-init\[2107]: Jan 12 14:52:40 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[112/120s]: request error \[HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 101] Network is unreachable',))]  
  
\[  127.528455] cloud-init\[2107]: Jan 12 14:52:47 cloud-init\[2107]: DataSourceEc2.py\[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno -2] Name or service not known',))  
  
\[  127.548962] cloud-init\[2107]: Jan 12 14:52:47 cloud-init\[2107]: url_helper.py\[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed \[119/120s]: unexpected error \[Attempted to set connect timeout to 0.0, but the timeout cannot be set to a value less than or equal to 0.]  
  
\[  134.556546] cloud-init\[2107]: Jan 12 14:52:54 cloud-init\[2107]: DataSourceEc2.py\[CRITICAL]: Giving up on md from \['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 126 seconds  
  
\[\[32m  OK  \[0m] Started Initial cloud-init job (metadata service crawler).  
  
Edited by: idogrady on Jan 12, 2020 8:40 AM

Answer

Hi Ido,  
  
On the log that you have provided, it can be seen, that networking (eth0) has not been configured correctly.  
  
To troubleshoot this kind of issue, I would recommend you to launch a new instance from the same AMI, and to attach the root volume from the broken instance i-096ae05732c55de39 to it. Then mount that root volume and compare its network configuration files with the configuration files of the working instance. As both instances are based on the same AMI, you should be able to find where the network is misconfigured easily.  
  
Regards,  
awstomas

Answer

Hi Tomas,  
  
Thank you for your help, however, I tried mounting the root volume to a similar instance which resulted in the new instance to have the same reachability error. I've also tried to add the root volume as an additional volume to a similar instance which and mount it which showed the newly mounted volume as "lost+found" which as I understand means that the volume is corrupted? In any case, I couldn't reach the mount volume data.  
  
Any help will be much appreciated.

EC2 Instance reachability check failed after yum upgrade

Relevant content