EC2 instance just stopped working. What happened?

Question

Recently our running EC2 instance (t2.micro) failed and we weren't able to login anymore. Even though in the EC2 console it had Online Ping Status.
So we had to Stop it and Start it again, which changed the IP address of the site, and caused some issues in our IoT online platform :(

To understand what happened, I checked for that EC2 instance reports with different metrics, and I see in the exact moment when it stopped working, the CPU usage increased suddenly to 100% and Network In Data also increased in a very unusual way. 
This is a very high peak druing a minute, in a whole year and several months that it had been working with no single issue.
Also the FailChecksum graph was high from that moment until we Stopped and Started the EC2 instance again.
Graphs here: 
https://ibb.co/qsgPXXs
https://ibb.co/YhpQHmQ
https://ibb.co/MSYmN8g

Do you have any idea of what could cause this anomaly?
And which would be the way to find out exactly?
Is is this a common issue?

Thank you in advance for any help.

Answer

As you've already found, an EC2 instance launched in a Public Subnet can be assigned a public IP address but the association only lasts as long as the instance is running. This is fine for testing/infrequent processing activities but not an advisable approach when running production/critical infrastructure. I would recommend you associate an Elastic IP address with your instance which will allow you to retain the same IP across any future restart/termination events.

It's also worth you reviewing the EC2 Best Practices to ensure your infrastructure is reliable and secure.

It's also worth you reviewing the [EC2 Best Practices](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-best-practices.html) to ensure your infrastructure is reliable and secure.

Answer

Have you tried using the [Amazon Cloudwatch agent ](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html) to acquire more information? You might also try using [Amazon Inspector](https://docs.aws.amazon.com/inspector/latest/user/what-is-inspector.html) if you suspect a malicious actor. Another option is to use an Application Performance Monitoring Tool if you cannot instrument your application, or [Amazon X-Ray](https://docs.aws.amazon.com/xray/latest/devguide/aws-xray.html), if you can instrument your code.

EC2 instance just stopped working. What happened?

相关内容