Why did Amazon EC2 Auto Scaling terminate an instance?

Short description

Amazon EC2 Auto Scaling relies on Amazon EC2 or Elastic Load Balancing (ELB) health checks to determine the health state of an instance. All the scaling actions of an Auto Scaling group, including health check replacements, are visible in the Activity History view on the Amazon EC2 console.

Resolution

Use the instance’s description in the Activity History view to determine further steps.

Before proceeding, find the description and cause for the instance’s termination:

1.    Open the Amazon EC2 console.

2.    In the navigation pane, under Auto Scaling, choose Auto Scaling Groups. Then, choose the instance’s group.

3.    Choose the Activity History view, and then choose the instance termination event.

4.    Note the Description and Cause for the instance termination event.

Refer to the following description examples to understand probable underlying reasons for instance termination.

"An instance was taken out of service in response to an EC2 health check indicating it has been terminated or stopped"

Find instance or system check failures with Amazon CloudWatch metrics:

1.    Open the Amazon CloudWatch console.

2.    In the navigation pane, choose Metrics, and then choose the All Metrics view.

3.    Choose EC2 in the metrics panel, and then choose Per-Instance Metrics.

4.    Type the instance-id, and then choose StatusCheckFailed_Instance, StatusCheckFailed_System, or StatusCheckFailed to view metrics graphs.

Amazon EC2 Auto Scaling terminates stopped and rebooted instances. Check AWS CloudTrail history to determine if a user manually stopped or rebooted the instance:

1.    Open the AWS CloudTrail console.

2.    In the navigation pane, choose Event history.

3.    Check the history for TerminateInstances, StopInstances, or RebootInstances API calls.

Amazon EC2 Auto Scaling terminates Spot instances when either of the following occurs:

• Capacity is no longer available
• Spot price exceeds the maximum price that you specified for the instances

Activity History might show that the instance was removed from service due to a health check. To verify the termination reason, check the Spot requests status:

1.    Open the Amazon EC2 console.

2.    In the navigation pane, under Instances, choose Spot Requests.

3.    Select the Spot request, choose the Description view, and then note the Status.

If an Auto Scaling group’s health check type is set to ELB, Amazon EC2 Auto Scaling performs both ELB and EC2 health checks. Then, the service terminates instances that fail either health check. To understand why an instance was terminated:

1.    Open the Amazon EC2 console.

2.    In the navigation pane, under Auto Scaling, choose Auto Scaling Groups, and then choose the instance's group.

3.    Choose the Details view, and then note the Health Check Type.

"An instance was taken out of service in response to a user request"

Review the CloudTrail events history during the period of time that the instance was taken out of service for specific API calls.

1.    Open the AWS CloudTrail console.

2.    In the navigation pane, choose Event history.

3.    Check the history for TerminateInstances, TerminateInstanceInAutoScalingGroup, or UpdateAutoScalingGroup API calls

"An instance was taken out of service in response to a ELB system health check failure"

For more information, see An instance was taken out of service in response to an ELB system health check failure.

If there are multiple load balancers attached to an Auto Scaling group, all load balancers must report an instance as healthy for Amazon EC2 Auto Scaling to consider the instance healthy.

1.    Open the Amazon EC2 console.

2.    In the navigation pane, under Load Balancing, choose Load Balancers, then choose Monitoring.

3.    Select the Health Hosts Count metric graph to confirm that the instance is failing health checks.

4.    In the navigation pane, under Auto Scaling, choose Auto Scaling Groups.

5.    Choose the instance’s group, and then choose the Details view.

6.    Note if there is more than one load balancer attached under Load Balancers or Target Groups.

Check CloudTrail history to determine if a suspended process delayed the termination of an unhealthy instance until the process resumed:

1.    Open the AWS CloudTrail console.

2.    In the navigation pane, choose Event history.

3.    Check the history for SuspendProcesses and ResumeProcesses API calls.

"An instance was taken out of service in response to a user health-check"

You can define custom health checks in Amazon EC2 Auto Scaling. When a custom health check determines that an instance is unhealthy, the check manually starts SetInstanceHealth and then sets the instance's state to Unhealthy. Amazon EC2 Auto Scaling terminates the unhealthy instance on its next run.

"An instance was launched to aid in balancing the group's zones" or "Instances were launched to balance instances in zones us-east-1a with other zones"

By default, Amazon EC2 Auto Scaling balances instances across all Availability Zones. When you add a new Availability Zone to an Auto Scaling group, Amazon EC2 Auto Scaling launches a new instance in that zone, and rebalancing might terminate instances in other zones.

"At 2018-02-12T13:48:46Z a monitor alarm XXX-High-CPU-Utilization in state ALARM triggered policy AAA-scaledown changing the desired capacity from 2 to 1"

Amazon EC2 Auto Scaling can terminate instances in a group in response to a configured CloudWatch alarm. Check the group policies and CloudWatch alarm history:

To check the Auto Scaling group policies:

1.    Open the Amazon EC2 console.

2.    In the navigation pane, under Auto Scaling, choose Auto Scaling Groups.

3.    Choose the instance’s group.

4.    Choose the Scaling Policies pane to view and edit policies.

To view the CloudWatch Alarm history:

1.    Open the Amazon CloudWatch console.

2.    In the navigation pane, choose Alarms.

3.    Choose the alarm, and then choose the History view.

4.    Check the history for any state changes to the alarm or modifications to the alarm configuration.

"At 2018-02-12T13:49:12Z a scheduled action update of AutoScalingGroup constraints to min: 1, max: 9, desired: 1 changing the desired capacity from 2 to 1. [...]"

You can configure scheduled actions that change an Auto Scaling group’s minimum, maximum, or desired capacity. When the scheduled action decreases the desired capacity, Amazon EC2 Auto Scaling terminates one or more instances to match the new desired capacity.

"A user request update of AutoScalingGroup constraints to min: 0, max: 2, desired: 1 changing the desired capacity from 2 to 1"

When you manually change the constraints of an Auto Scaling group, such as reducing the desired capacity, Amazon EC2 Auto Scaling can terminate instances to match the new settings.

---

Related information

Why did my Auto Scaling group scale down?

Why didn’t Amazon EC2 Auto Scaling terminate an unhealthy instance?