My EC2 Status Checks fails, why this happens?

0

Hi, I deployed a web server on ec2 machine(ubuntu 18.04), t2.micro(instace type). web and database server is running under Docker. But last night ec2 status check was failed.
Then i reboot the instance and look at the syslogs,
I found below logs:-

  1. Network configuration changed, trying to establish connection.
    Synchronized to time server 91.189.91.157:123 (ntp.ubuntu.com).

  2. ERROR Health ping failed with error - EC2RoleRequestError: no EC2 instance role found.
    caused by: EC2MetadataError: failed to make EC2Metadata request

caused by:

  <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
#011"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
 <title>404 - Not Found</title>
  </head>
 <body>
<h1>404 - Not Found</h1>
  </body>
 </html
>```

**Can anyone suggest me whats wrong with my ec2-machine?**
gefragt vor 4 Jahren2253 Aufrufe
2 Antworten
0

I am finding the same log entries. This began when normal status checks began failing with this notation on EC2 status: Instance reachability check failed at February 12, 2020 at 9:31:00 PM UTC-8 (38 days, 17 hours and 52 minutes ago)

Here is a recent syslog entry that shows the sequence:

Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: 2020-03-22 23:59:18 INFO Backing off health check to every 1200 seconds for 3600 seconds.
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: 2020-03-22 23:59:18 ERROR Health ping failed with error - EC2RoleRequestError: no EC2 instance role found
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: caused by: EC2MetadataError: failed to make EC2Metadata request
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: caused by: <?xml version="1.0" encoding="iso-8859-1"?>
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: #011"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]:  <head>
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]:   <title>404 - Not Found</title>
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]:  </head>
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]:  <body>
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]:   <h1>404 - Not Found</h1>
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]:  </body>
Mar 22 23:59:18 ip-172-31-17-50 amazon-ssm-agent.amazon-ssm-agent[1782]: </html>

AFAICT, the instance was running normally before this time. It was initially turned up Wed Jan 29 06:17:35 UTC 2020

What did AWS change at that time to cause this error, which took down my server?

DVHirst865

beantwortet vor 4 Jahren
0

Sadly I don't have an answer for you but I wanted to note that the EC2RoleRequestError may well be a red-herring.

On a fresh Ubuntu 20.04 instance running on a T4g.micro, the syslog contains the same error you're seeing on the first (and every) boot:

Oct  5 15:19:28 ip-172-31-10-32 systemd[1]: Started Service for snap application amazon-ssm-agent.amazon-ssm-agent.
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: 2020/10/05 15:19:28 Failed to load instance info from vault. RegistrationKey does not exist.
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: Error occurred fetching the seelog config file path:  open /etc/amazon/ssm/seelog.xml: no such file or directory
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: Initializing new seelog logger
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: New Seelog Logger Creation Complete
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: 2020-10-05 15:19:28 ERROR Error adding the directory to watcher: no such file or directory
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: 2020-10-05 15:19:28 INFO Agent is in hibernate mode. Reducing logging. Logging will be reduced to one log per backoff period
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: 2020-10-05 15:19:28 INFO Entering SSM Agent hibernate - EC2RoleRequestError: no EC2 instance role found
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: caused by: EC2MetadataError: failed to make EC2Metadata request
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: #011status code: 404, request id:
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: caused by: <?xml version="1.0" encoding="iso-8859-1"?>
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: #011#011 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]:  <head>
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]:   <title>404 - Not Found</title>
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]:  </head>
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]:  <body>
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]:   <h1>404 - Not Found</h1>
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]:  </body>
Oct  5 15:19:28 ip-172-31-10-32 amazon-ssm-agent.amazon-ssm-agent[1757]: </html>

It also continues to appear after updating all packages to their latest versions.

This doesn't seem to actually prevent the metrics working though, although I have noticed that the disk activity metrics are empty.
(I found these logs after I went digging to figure out why startup was taking so long)

Presumably something inside AWS is broken (or somehow incompatible with Ubuntu) but unless your alerts are disk-related (or maybe using the enhanced monitoring?), I don't think this is the cause.

Edited by: davidje13 on Oct 5, 2020 8:33 AM

beantwortet vor 4 Jahren

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen