My EC2 tg4.micro simply disappeared today with no trace

0

Hello,

My EC2 tg4.micro simply vanished today, until yesterday night I was accessing it fine.

This instance was running fine since 23 July, Region us-east-1 (N. Virginia), with AMI Amazon Linux 2 arm64, and almost vanilla with nothing fancy installed yet.

Nothing new or scheduled was On, The EBS is gone too, no trace.

Uniq tip I have is my cloud watch cpu alarm(credit balance spot), which stopped receiving data today at 3 AM (UTC), the EC2 Events have no logs.

I have 2 security groups on it, internal user ec2-user disable, my user was using ssh pairs so still no password ssh login allowed...

I use AWS for a couple of years (since 2018) and have never experienced something like that, I am with no clue here.

Thanks in advance!

brunof
demandé il y a 3 ans525 vues
4 réponses
0

Well that's terrifying! I don't have an answer sadly. Out of curiosity, was it a spot instance?

répondu il y a 3 ans
0

Me too =/ ...Never thought that this kind of thing could happen with no trace...

Yup, it was a spot instance, but even if it was the case that my credits expire or reached some kind of limit, my instance should be stopped but never deleted and I have no policy/role for that so...I even lookup Spot Instance pricing history for any changes but its completely stable for t4gmicro in that region...

I already opened a support case because I still have hope that inside AWS they can find what happen reading logs that customers don't have access to.

brunof
répondu il y a 3 ans
0

brunof wrote:
Me too =/ ...Never thought that this kind of thing could happen with no trace...

Yup, it was a spot instance, but even if it was the case that my credits expire or reached some kind of limit, my instance should be stopped but never deleted and I have no policy/role for that so...I even lookup Spot Instance pricing history for any changes but its completely stable for t4gmicro in that region...

I already opened a support case because I still have hope that inside AWS they can find what happen reading logs that customers don't have access to.
Hello, just an update here.

It really was an EC2 Spot related cause, when I though interrupted I didn't assimilate it as erased =P

I just received a helpful reply from AWS support.

So I probably set a “maximum Spot price" when creating the instance, and it was claimed and terminated after a 2-minute notification(which I thought was an email).

See more:
You can read more about Spot Instance requests and terminations in the "Understanding Spot Instances and Spot Price" section of our Spot Instances page:

http://aws.amazon.com/ec2/spot-instances

The EBS cause was also explained by the AWS:
When a Spot instance is terminated:

1) All data on instance-storage will be lost.
2) All EBS volumes that are attached which are set to "Delete on Terminate" will be deleted.
3) All EBS volumes that are attached which are set to NOT "Delete on Terminate" will be left unattached to any instance. You can then attach them to another instance and get to your data.

So, my mistake folks, still faithful using AWS again ;)

brunof
répondu il y a 3 ans
0

brunof wrote:
Me too =/ ...Never thought that this kind of thing could happen with no trace...

Yup, it was a spot instance, but even if it was the case that my credits expire or reached some kind of limit, my instance should be stopped but never deleted and I have no policy/role for that so...I even lookup Spot Instance pricing history for any changes but its completely stable for t4gmicro in that region...

I already opened a support case because I still have hope that inside AWS they can find what happen reading logs that customers don't have access to.
Hello, final update here.

Yesterday I found the log at CloudTrails -> Event history with one entry with name BidEvictedEvent

By default the behavior is to terminate the Spot instance, checked on cli with:
aws ec2 describe-spot-instance-requests

Useful link:
[Why is my Spot Instance terminating even though the Spot price doesn't exceed the maximum price?]:
https://aws.amazon.com/pt/premiumsupport/knowledge-center/ec2-spot-instance-termination-reasons/

[Specify the interruption behavior]
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html#specifying-spot-interruption-behavior

Edited by: brunof on Aug 6, 2021 6:28 AM

brunof
répondu il y a 3 ans

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions