My EC2 tg4.micro simply disappeared today with no trace

0

Hello,

My EC2 tg4.micro simply vanished today, until yesterday night I was accessing it fine.

This instance was running fine since 23 July, Region us-east-1 (N. Virginia), with AMI Amazon Linux 2 arm64, and almost vanilla with nothing fancy installed yet.

Nothing new or scheduled was On, The EBS is gone too, no trace.

Uniq tip I have is my cloud watch cpu alarm(credit balance spot), which stopped receiving data today at 3 AM (UTC), the EC2 Events have no logs.

I have 2 security groups on it, internal user ec2-user disable, my user was using ssh pairs so still no password ssh login allowed...

I use AWS for a couple of years (since 2018) and have never experienced something like that, I am with no clue here.

Thanks in advance!

brunof
asked 3 years ago511 views
4 Answers
0

Well that's terrifying! I don't have an answer sadly. Out of curiosity, was it a spot instance?

answered 3 years ago
0

Me too =/ ...Never thought that this kind of thing could happen with no trace...

Yup, it was a spot instance, but even if it was the case that my credits expire or reached some kind of limit, my instance should be stopped but never deleted and I have no policy/role for that so...I even lookup Spot Instance pricing history for any changes but its completely stable for t4gmicro in that region...

I already opened a support case because I still have hope that inside AWS they can find what happen reading logs that customers don't have access to.

brunof
answered 3 years ago
0

brunof wrote:
Me too =/ ...Never thought that this kind of thing could happen with no trace...

Yup, it was a spot instance, but even if it was the case that my credits expire or reached some kind of limit, my instance should be stopped but never deleted and I have no policy/role for that so...I even lookup Spot Instance pricing history for any changes but its completely stable for t4gmicro in that region...

I already opened a support case because I still have hope that inside AWS they can find what happen reading logs that customers don't have access to.
Hello, just an update here.

It really was an EC2 Spot related cause, when I though interrupted I didn't assimilate it as erased =P

I just received a helpful reply from AWS support.

So I probably set a “maximum Spot price" when creating the instance, and it was claimed and terminated after a 2-minute notification(which I thought was an email).

See more:
You can read more about Spot Instance requests and terminations in the "Understanding Spot Instances and Spot Price" section of our Spot Instances page:

http://aws.amazon.com/ec2/spot-instances

The EBS cause was also explained by the AWS:
When a Spot instance is terminated:

1) All data on instance-storage will be lost.
2) All EBS volumes that are attached which are set to "Delete on Terminate" will be deleted.
3) All EBS volumes that are attached which are set to NOT "Delete on Terminate" will be left unattached to any instance. You can then attach them to another instance and get to your data.

So, my mistake folks, still faithful using AWS again ;)

brunof
answered 3 years ago
0

brunof wrote:
Me too =/ ...Never thought that this kind of thing could happen with no trace...

Yup, it was a spot instance, but even if it was the case that my credits expire or reached some kind of limit, my instance should be stopped but never deleted and I have no policy/role for that so...I even lookup Spot Instance pricing history for any changes but its completely stable for t4gmicro in that region...

I already opened a support case because I still have hope that inside AWS they can find what happen reading logs that customers don't have access to.
Hello, final update here.

Yesterday I found the log at CloudTrails -> Event history with one entry with name BidEvictedEvent

By default the behavior is to terminate the Spot instance, checked on cli with:
aws ec2 describe-spot-instance-requests

Useful link:
[Why is my Spot Instance terminating even though the Spot price doesn't exceed the maximum price?]:
https://aws.amazon.com/pt/premiumsupport/knowledge-center/ec2-spot-instance-termination-reasons/

[Specify the interruption behavior]
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html#specifying-spot-interruption-behavior

Edited by: brunof on Aug 6, 2021 6:28 AM

brunof
answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions