My EC2 tg4.micro simply disappeared today with no trace

0

Hello,

My EC2 tg4.micro simply vanished today, until yesterday night I was accessing it fine.

This instance was running fine since 23 July, Region us-east-1 (N. Virginia), with AMI Amazon Linux 2 arm64, and almost vanilla with nothing fancy installed yet.

Nothing new or scheduled was On, The EBS is gone too, no trace.

Uniq tip I have is my cloud watch cpu alarm(credit balance spot), which stopped receiving data today at 3 AM (UTC), the EC2 Events have no logs.

I have 2 security groups on it, internal user ec2-user disable, my user was using ssh pairs so still no password ssh login allowed...

I use AWS for a couple of years (since 2018) and have never experienced something like that, I am with no clue here.

Thanks in advance!

brunof
질문됨 3년 전525회 조회
4개 답변
0

Well that's terrifying! I don't have an answer sadly. Out of curiosity, was it a spot instance?

답변함 3년 전
0

Me too =/ ...Never thought that this kind of thing could happen with no trace...

Yup, it was a spot instance, but even if it was the case that my credits expire or reached some kind of limit, my instance should be stopped but never deleted and I have no policy/role for that so...I even lookup Spot Instance pricing history for any changes but its completely stable for t4gmicro in that region...

I already opened a support case because I still have hope that inside AWS they can find what happen reading logs that customers don't have access to.

brunof
답변함 3년 전
0

brunof wrote:
Me too =/ ...Never thought that this kind of thing could happen with no trace...

Yup, it was a spot instance, but even if it was the case that my credits expire or reached some kind of limit, my instance should be stopped but never deleted and I have no policy/role for that so...I even lookup Spot Instance pricing history for any changes but its completely stable for t4gmicro in that region...

I already opened a support case because I still have hope that inside AWS they can find what happen reading logs that customers don't have access to.
Hello, just an update here.

It really was an EC2 Spot related cause, when I though interrupted I didn't assimilate it as erased =P

I just received a helpful reply from AWS support.

So I probably set a “maximum Spot price" when creating the instance, and it was claimed and terminated after a 2-minute notification(which I thought was an email).

See more:
You can read more about Spot Instance requests and terminations in the "Understanding Spot Instances and Spot Price" section of our Spot Instances page:

http://aws.amazon.com/ec2/spot-instances

The EBS cause was also explained by the AWS:
When a Spot instance is terminated:

1) All data on instance-storage will be lost.
2) All EBS volumes that are attached which are set to "Delete on Terminate" will be deleted.
3) All EBS volumes that are attached which are set to NOT "Delete on Terminate" will be left unattached to any instance. You can then attach them to another instance and get to your data.

So, my mistake folks, still faithful using AWS again ;)

brunof
답변함 3년 전
0

brunof wrote:
Me too =/ ...Never thought that this kind of thing could happen with no trace...

Yup, it was a spot instance, but even if it was the case that my credits expire or reached some kind of limit, my instance should be stopped but never deleted and I have no policy/role for that so...I even lookup Spot Instance pricing history for any changes but its completely stable for t4gmicro in that region...

I already opened a support case because I still have hope that inside AWS they can find what happen reading logs that customers don't have access to.
Hello, final update here.

Yesterday I found the log at CloudTrails -> Event history with one entry with name BidEvictedEvent

By default the behavior is to terminate the Spot instance, checked on cli with:
aws ec2 describe-spot-instance-requests

Useful link:
[Why is my Spot Instance terminating even though the Spot price doesn't exceed the maximum price?]:
https://aws.amazon.com/pt/premiumsupport/knowledge-center/ec2-spot-instance-termination-reasons/

[Specify the interruption behavior]
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html#specifying-spot-interruption-behavior

Edited by: brunof on Aug 6, 2021 6:28 AM

brunof
답변함 3년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠