My EC2 Status Checks fails, why this happens?

Question

Hi, I deployed a web server on ec2 machine(ubuntu 18.04), t2.micro(instace type). web and database server is running under Docker. But last night ec2 status check was failed. Then i reboot the instance and look at the syslogs, I found below logs:- 1. Network configuration changed, trying to establish connection. Synchronized to time server 91.189.91.157:123 (ntp.ubuntu.com). 2. ERROR Health ping failed with error - EC2RoleRequestError: no EC2 instance role found. caused by: EC2MetadataError: failed to make EC2Metadata request **caused by**: ``` 404 - Not Found

404 - Not Found

``` **Can anyone suggest me whats wrong with my ec2-machine?**

Answer

I am finding the same log entries. This began when normal status checks began failing with this notation on EC2 status: Instance reachability check failed at February 12, 2020 at 9:31:00 PM UTC-8 (38 days, 17 hours and 52 minutes ago)

Here is a recent syslog entry that shows the sequence:

Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: 2020-03-22 23:59:18 INFO Backing off health check to every 1200 seconds for 3600 seconds.
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: 2020-03-22 23:59:18 ERROR Health ping failed with error - EC2RoleRequestError: no EC2 instance role found
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: caused by: EC2MetadataError: failed to make EC2Metadata request
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: caused by: <?xml version="1.0" encoding="iso-8859-1"?>
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: #011"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]:  <head>
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]:   <title>404 - Not Found</title>
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]:  </head>
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]:  <body>
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]:   <h1>404 - Not Found</h1>
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]:  </body>
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: </html>

AFAICT, the instance was running normally before this time. It was initially turned up Wed Jan 29 06:17:35 UTC 2020

What did AWS change at that time to cause this error, which took down my server?

DVHirst865

404 - Not Found

Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: ``` AFAICT, the instance was running normally before this time. It was initially turned up Wed Jan 29 06:17:35 UTC 2020 What did AWS change at that time to cause this error, which took down my server? DVHirst865

Answer

Sadly I don't have an answer for you but I wanted to note that the EC2RoleRequestError may well be a red-herring.

On a fresh Ubuntu 20.04 instance running on a T4g.micro, the syslog contains the same error you're seeing on the first (and every) boot:

Oct  5 15:19:28 ip-172-31-10-32 systemd[1]: Started Service for snap application amazon-ssm-agent.amazon-ssm-agent.
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: 2020/10/05 15:19:28 Failed to load instance info from vault. RegistrationKey does not exist.
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: Error occurred fetching the seelog config file path:  open /etc/amazon/ssm/seelog.xml: no such file or directory
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: Initializing new seelog logger
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: New Seelog Logger Creation Complete
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: 2020-10-05 15:19:28 ERROR Error adding the directory to watcher: no such file or directory
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: 2020-10-05 15:19:28 INFO Agent is in hibernate mode. Reducing logging. Logging will be reduced to one log per backoff period
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: 2020-10-05 15:19:28 INFO Entering SSM Agent hibernate - EC2RoleRequestError: no EC2 instance role found
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: caused by: EC2MetadataError: failed to make EC2Metadata request
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: #011status code: 404, request id:
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: caused by: <?xml version="1.0" encoding="iso-8859-1"?>
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: #011#011 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]:  <head>
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]:   <title>404 - Not Found</title>
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]:  </head>
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]:  <body>
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]:   <h1>404 - Not Found</h1>
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]:  </body>
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: </html>

It also continues to appear after updating all packages to their latest versions.

This doesn't seem to actually prevent the metrics working though, although I have noticed that the disk activity metrics are empty.
(I found these logs after I went digging to figure out why startup was taking so long)

Presumably something inside AWS is broken (or somehow incompatible with Ubuntu) but unless your alerts are disk-related (or maybe using the enhanced monitoring?), I don't think this is the cause.

Edited by: davidje13 on Oct 5, 2020 8:33 AM

404 - Not Found

Oct 5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: Oct 5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: ``` It also continues to appear after updating all packages to their latest versions. This doesn't seem to actually prevent the metrics working though, although I have noticed that the disk activity metrics are empty. (I found these logs after I went digging to figure out why startup was taking so long) Presumably something inside AWS is broken (or somehow incompatible with Ubuntu) but unless your alerts are disk-related (or maybe using the enhanced monitoring?), I don't think this is the cause. Edited by: davidje13 on Oct 5, 2020 8:33 AM

My EC2 Status Checks fails, why this happens?

Relevant content