Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Issues with Spot instances

0

Hi,

For the past 15 days or so, I've been noticing abnormal behavior with the spot instances. Every day except weekends, between 4:00 PM and 12:00 PM approximately, machines are terminated en masse, in a loop because of a lack of Spot capacity. They rarely stay up for more than an hour, which is problematic. This problem occurs on us-east-1 (all zones) with c6i instances. I've tried changing the price to pay the same price as normal EC2 machines but it makes no difference.

I've been using spot machines for over 7 years on us-east-1 and I've never seen anything like this. What can I do?

  • Gildas
    2 years ago

    Minor correction, it's between 12:00 PM and 08:00 PM (GMT).

Topics
Compute
Tags
Amazon EC2Amazon EC2 Spot Instances
Language
English

asked 2 years ago1.3K views

2 Answers
1
Accepted Answer

Spot instances can offer significant cost savings, but they come with the inherent risk of being interrupted when the capacity is no longer available or the Spot price exceeds your bid price. The behavior you're describing suggests a sudden and significant increase in demand for c6i instances in the us-east-1 region, leading to frequent interruptions.

Reasons why they are dying may be just because there is a major increase in requests of spots such as the price is raising quicker than before, surpassing your bid. Another reason may be that they are making up for missing onDemand machines so AWS would take from the spot pool.

I would suggest you review your business case on using spot instances or wait to see if it changes.

EXPERT

answered 2 years ago

EXPERT

reviewed 2 years ago

EXPERT

reviewed 2 years ago

AWS
EXPERT

reviewed 2 years ago

  • Gildas
    2 years ago

    Thanks but my business case was OK for years. It just extremely changed suddenly 15 days ago, as I explained. I was hoping some people at AWS would be able to look a this issue because I can say that spot instances (c6i.large) on us-east-1 are totally useless between 12:00 PM and 08:00 PM (GMT) when this issue occurs.

  • Giovanni Lauria EXPERT
    2 years ago

    As I said, there may be a growth in requests AWS has got that pushed them not to being able to fulfill all the spot requests. The overall business justification of Spot Instances is being used for interruptible processes that you would not notice about in the short time. Think about Spot Pools or build custom ones if that is retired.

  • Isaac Behrens EXPERT
    2 years ago

    Spot is spare capacity. When it is needed elsewhere to meet on-demand, it is no longer spare and the spot instances will be terminated.

    Can't say specifically the cause, but Prime Day was not to long ago, which uses lots of infrastructure and this past weekend you had the Crowdstrike issue which is causing many to have to create temporary instances as part of their recovery, both of which means there will be higher demand and less spare capacity.

  • Gildas
    2 years ago

    Thank you for your additional information. It's probably due to something in the news as you suggested. Today it looks more stable. I'm keeping my fingers crossed as the spot instances have always suited me so far.

1

Hello,

There could be multiple reasons that can lead to termination of spot instances. You can refer to this knowledge center post to find out why the Spot Instances are getting terminated

Please refer to best practices for EC2 Spot that you leverage to overcome your issue.

AWS

answered 2 years ago

EXPERT

reviewed 2 years ago

  • Gildas
    2 years ago

    Thank you.

    From my point of view, a machine should last at least 1 hour, which is not the case. I will try to use other types of instances to get around this problem if it continues. It's silly because it was the AWS support team that advised me to use c6i instances in the first place...

    I would have hoped that Amazon would also look into this problem, which seems to be global at first sight.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content