Spot instances not terminating

Question

I have set up an ASG and deployed an auto-spotting AMI using CloudFormation and StackSets. The transition from On-Demand to Spot Instances is working as expected.
However, I'm not receiving EC2 Spot Instance interruption notices. To verify the setup, I ran an AWS FIS experiment that triggered Spot Instance interruptions. The ALB successfully handled traffic without causing application downtime, which confirms the resilience of the setup.

That said, I'm concerned about the absence of EC2 interruption notices. Is there a reason why the interruption notifications are not being received?

re:Post Agent · Answer

There could be a few reasons why you're not receiving EC2 Spot Instance interruption notices:

1. Configuration: Ensure that you have properly configured Amazon EventBridge to capture and process these interruption notices. The notices are emitted as events two minutes prior to the actual interruption (except for hibernation, which occurs immediately).

2. Interruption behavior: Check the interruption behavior you've specified for your Spot Instances. If you've configured them to stop or hibernate instead of terminate, the behavior might differ from what you're expecting.

3. Constraints: If your Spot request includes constraints such as launch groups or Availability Zone groups, instances might be terminated as a group when the constraint can no longer be met, potentially affecting how interruption notices are handled.

4. Auto Scaling Group (ASG) behavior: Your ASG might be handling the interruptions automatically, replacing instances before you notice the interruption. This could explain why your application remained resilient during the AWS FIS experiment.

5. Instance lifecycle: Verify that the instances in question are actually Spot Instances. You can do this by filtering for "Instance lifecycle=spot" in the EC2 console.

6. Interruption reason: Some interruptions might occur without the standard two-minute notice, depending on the reason for the interruption.

To troubleshoot this issue:

1. Double-check your EventBridge configuration to ensure it's set up to capture EC2 Spot Instance Interruption Warning events.

2. Review the instance metadata for your Spot Instances to see if the 'instance-action' item is present, which indicates an impending interruption.

3. Check CloudTrail logs for 'BidEvictedEvent' entries, which would indicate that EC2 terminated a Spot Instance.

4. Verify your ASG and Spot Fleet configurations to ensure they're not masking the interruption process.

If after these checks you're still not seeing interruption notices, it might be worth contacting AWS support for further investigation, as the notices should be emitted under normal circumstances when Spot Instances are about to be interrupted.

**Sources**  
[Determine whether Amazon EC2 terminated a Spot Instance - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/BidEvictedEvent.html)  
[Managing Spot Instance Interruptions - Overview of Amazon EC2 Spot Instances](https://docs.aws.amazon.com/whitepapers/latest/cost-optimization-leveraging-ec2-spot-instances/managing-instance-termination.html)  
[Spot Instance interruptions - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html)  
[Spot Instance interruption notices - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html)

Spot instances not terminating

Relevant content