By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

ec2 instance reachability issue - "Instance reachability check failed"

0

I recently performed a yum update on my server running Amazon Linux 2. Ever since then, I cannot connect to the server, and it is failing the status check with the message "Instance reachability check failed." I have read through many posts about this, but I can't seem to find a solution, or the posts asking similar questions are never answered.

I did find some information about ip address "169.254.169.254", which shows up in the log. Someone said something about network issue? But I don't understand why that would be an issue for me, I did not change any of that. I just did the update and rebooted?

Here is a portion of my system log, I am not sure what else to provide.

         Starting Postfix Mail Transport Agent...

[    7.273092] cloud-init[2237]: Cloud-init v. 19.3-46.amzn2.0.1 running 'init' at Mon, 05 Jun 2023 17:19:53 +0000. Up 7.18 seconds.

[    7.304066] cloud-init[2237]: ci-info: +++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++

[    7.304323] cloud-init[2237]: ci-info: +--------+------+---------------------------+-----------+-------+-------------------+

[    7.304448] cloud-init[2237]: ci-info: | Device |  Up  |          Address          |    Mask   | Scope |     Hw-Address    |

[    7.304549] cloud-init[2237]: ci-info: +--------+------+---------------------------+-----------+-------+-------------------+

[    7.304644] cloud-init[2237]: ci-info: |  eth0  | True | fe80::12:2ff:fe51:120b/64 |     .     |  link | 02:12:02:51:12:0b |

[    7.304774] cloud-init[2237]: ci-info: |   lo   | True |         127.0.0.1         | 255.0.0.0 |  host |         .         |

[    7.304901] cloud-init[2237]: ci-info: |   lo   | True |          ::1/128          |     .     |  host |         .         |

[    7.305004] cloud-init[2237]: ci-info: +--------+------+---------------------------+-----------+-------+-------------------+

[    7.305108] cloud-init[2237]: ci-info: ++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++

[    7.305207] cloud-init[2237]: ci-info: +-------+-----------------+---------+-----------------+-----------+-------+

[    7.305322] cloud-init[2237]: ci-info: | Route |   Destination   | Gateway |     Genmask     | Interface | Flags |

[    7.305450] cloud-init[2237]: ci-info: +-------+-----------------+---------+-----------------+-----------+-------+

[    7.305567] cloud-init[2237]: ci-info: |   0   | 169.254.169.254 | 0.0.0.0 | 255.255.255.255 |    eth0   |   UH  |

[    7.305673] cloud-init[2237]: ci-info: +-------+-----------------+---------+-----------------+-----------+-------+

[    7.305767] cloud-init[2237]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++

[    7.305864] cloud-init[2237]: ci-info: +-------+-------------+---------+-----------+-------+

[    7.305964] cloud-init[2237]: ci-info: | Route | Destination | Gateway | Interface | Flags |

[    7.306059] cloud-init[2237]: ci-info: +-------+-------------+---------+-----------+-------+

[    7.306187] cloud-init[2237]: ci-info: |   9   |  fe80::/64  |    ::   |    eth0   |   U   |

[    7.306317] cloud-init[2237]: ci-info: |   11  |    local    |    ::   |    eth0   |   U   |

[    7.306429] cloud-init[2237]: ci-info: |   12  |   ff00::/8  |    ::   |    eth0   |   U   |

[    7.306553] cloud-init[2237]: ci-info: +-------+-------------+---------+-----------+-------+

[  OK  ] Started Postfix Mail Transport Agent.

[  OK  ] Started MariaDB database server.

[  OK  ] Stopped Run the CVE-2021-44228 hotfix script.

[  OK  ] Started Run the CVE-2021-44228 hotfix script.

[   57.616696] cloud-init[2237]: Jun 05 17:20:44 cloud-init[2237]: url_helper.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [50/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7f7f4ba27dd0>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)'))]

[  108.669221] cloud-init[2237]: Jun 05 17:21:35 cloud-init[2237]: url_helper.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [101/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7f7f4ba24a10>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)'))]

[  126.689596] cloud-init[2237]: Jun 05 17:21:53 cloud-init[2237]: url_helper.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [119/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7f7f4ba249d0>, 'Connection to 169.254.169.254 timed out. (connect timeout=17.0)'))]

[  127.690978] cloud-init[2237]: Jun 05 17:21:54 cloud-init[2237]: DataSourceEc2.py[WARNING]: IMDS's HTTP endpoint is probably disabled

[  OK  ] Started Initial cloud-init job (metadata service crawler).

[  OK  ] Reached target Cloud-config availability.

[  OK  ] Reached target Network is Online.

         Starting Apply the settings specified in cloud-config...

[  OK  ] Started amazon-ssm-agent.

         Starting Dynamically Generate Message Of The Day...

         Starting System Logging Service...

         Starting Notify NFS peers of a restart...

         Starting OpenSSH server daemon...

         Starting Permit User Sessions...

[  OK  ] Started Notify NFS peers of a restart.

[  OK  ] Started Permit User Sessions.

[  OK  ] Started Command Scheduler.

         Starting Terminate Plymouth Boot Screen...

         Starting Wait for Plymouth Boot Screen to Quit...

[  OK  ] Started Job spooling tools.

[  OK  ] Created slice User Slice of root.

[FAILED] Failed to start Session c1 of user root.

See 'systemctl status session-c1.scope' for details.

[FAILED] Failed to start Session c3 of user root.

See 'systemctl status session-c3.scope' for details.

[FAILED] Failed to start Session c2 of user root.

See 'systemctl status session-c2.scope' for details.

[  OK  ] Started System Logging Service.

[  OK  ] Started OpenSSH server daemon.

[  OK  ] Removed slice User Slice of root.

[  128.602842] cloud-init[3263]: Cloud-init v. 19.3-46.amzn2.0.1 running 'modules:config' at Mon, 05 Jun 2023 17:21:54 +0000. Up 128.53 seconds.
[  129.036365] cloud-init[3322]: Cloud-init v. 19.3-46.amzn2.0.1 running 'modules:final' at Mon, 05 Jun 2023 17:21:55 +0000. Up 128.98 seconds.
[  129.050401] cloud-init[3322]: Cloud-init v. 19.3-46.amzn2.0.1 finished at Mon, 05 Jun 2023 17:21:55 +0000. Datasource DataSourceNone.  Up 129.04 seconds
[  129.057795] cloud-init[3322]: Jun 05 17:21:55 cloud-init[3322]: cc_final_message.py[WARNING]: Used fallback datasource


Amazon Linux 2
Kernel 4.14.314-238.539.amzn2.x86_64 on an x86_64

ip-172-30-0-37 login: [  259.040541] hibinit-agent[3321]: Traceback (most recent call last):
[  259.040995] hibinit-agent[3321]: File "/usr/bin/hibinit-agent", line 496, in <module>
[  259.041502] hibinit-agent[3321]: main()
[  259.042012] hibinit-agent[3321]: File "/usr/bin/hibinit-agent", line 435, in main
[  259.050239] hibinit-agent[3321]: if not hibernation_enabled(config.state_dir):
[  259.050517] hibinit-agent[3321]: File "/usr/bin/hibinit-agent", line 390, in hibernation_enabled
[  259.056646] hibinit-agent[3321]: imds_token = get_imds_token()
[  259.056927] hibinit-agent[3321]: File "/usr/bin/hibinit-agent", line 365, in get_imds_token
[  259.057515] hibinit-agent[3321]: response = requests.put(token_url, headers=request_header)
[  259.058573] hibinit-agent[3321]: File "/usr/lib/python2.7/site-packages/requests/api.py", line 121, in put
[  259.074207] hibinit-agent[3321]: return request('put', url, data=data, **kwargs)
[  259.074529] hibinit-agent[3321]: File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
[  259.074993] hibinit-agent[3321]: response = session.request(method=method, url=url, **kwargs)
[  259.075492] hibinit-agent[3321]: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 486, in request
[  259.077613] hibinit-agent[3321]: resp = self.send(prep, **send_kwargs)
[  259.078188] hibinit-agent[3321]: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 598, in send
[  259.079262] hibinit-agent[3321]: r = adapter.send(request, **kwargs)
[  259.080340] hibinit-agent[3321]: File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 419, in send
[  259.088082] hibinit-agent[3321]: raise ConnectTimeout(e, request=request)
[  259.088484] hibinit-agent[3321]: requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7ff5e5b3aa90>: Failed to establish a new connection: [Errno 110] Connection timed out',))
Topics
Compute
Tags
Amazon EC2Amazon Linux
Language
English
rePost-User-8371476
asked 10 months ago880 views
4 Answers
0

Hi, IMHO, the issue may come from 0.0.0.0 as gateway: it means no gateway, which means then that you cannot connect from any remote machine. Maybe the yum update messed up the network.

What I would to try to recover is start a new EC2 instance on same subnet, connect to this new instance from remote. Then, when you are connected, try a ssh connection to your problematic machine. Since you will be on same subnet, gateway won't be necessary and you may have a chance to connect to your failing machine to reconfigure the gateway.

Good luck!

profile pictureAWS
EXPERT
Didier_Durand
answered 10 months ago
  • Didier_Durand EXPERT
    10 months ago

    Did you try to ping your problematic instance from one of the 2 others? (to validate that network stack works from local subnet at least up to ICMP layer)

0

Thanks @Didier_AWS, I actually have two other instances running on that same subnet and I could SSH from one to the other and vice-versa using the internal the ip address, but I could not get to my instance with the failing test?

Oddly, and maybe this does not mean anything, when I ran "Get instance screenshot", I got this. Which seems to show the system is running, maybe? So that sort of goes with your ideal of my instance having a network issue of some kind?

Enter image description here

I also tried connecting to "EC2 serial console" which came up as an option and it just sat there with a blank screen?

rePost-User-8371476
answered 10 months ago
  • Didier_Durand EXPERT
    10 months ago

    See my new comment in my answer.

0

169.254.169.254 is the address to access the EC2 instance metadata https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html this is normal and won't be the cause of your problem.

The "Net device info" at the top of your system log only has an IPv6 address associated with it, there isn't any IPv4 listed (that's not to say there isn't an IPv4 address set, and it's just not captured in these logs).

In AWS console do you have an IPv4 address (either public or private) associated with this instance (from the screengrab I can guess that it's likely to be 172.30.0.37 )?

You mention you've got two other instances in the same subnet and neither can SSH to the problem instance, well can they get any life out of it on IPv6? e.g. can you open up the security group to allow inbound ICMPv6 and IPv6 port 22 and just see if it responds to a ping or attempt to SSH on the IPv6 address?

That might get you something to work with, or at least going in the right direction.

profile picture
EXPERT
Steve_M
answered 10 months ago
0

Thanks for the feedback and suggestions. I was unsuccessful with the ip6 and pinging the server. I ended up just mounting the drive on a different server and I copied my files over to a new server.

I have had multiple servers on AWS for years and I have never had an issue with an upgrade like this. Oh well, I guess I will be backing up prior to upgrades in future.

rePost-User-8371476
answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content