Why did Amazon EC2 Auto Scaling terminate an instance?

Short description

Amazon EC2 Auto Scaling monitors your instance's health status with Amazon EC2 health checks, Elastic Load Balancing (ELB) health checks, and volume checks. If your instance doesn't pass a check, then Amazon EC2 Auto Scaling terminates and replaces your instance.

Review why Amazon EC2 Auto Scaling terminated your instance in the Amazon EC2 dashboard in the AWS Management Console. Then, troubleshoot the issue that you find.

Resolution

Verify why your instance failed its health check, and then troubleshoot based on the reason for your health check failure.

Troubleshoot instances that Amazon EC2 Auto Scaling terminated or stopped

Amazon EC2 Auto Scaling terminates stopped and rebooted instances. The following message indicates that Amazon EC2 Auto Scaling stopped, rebooted, or terminated an instance:

"An instance was taken out of service in response to an EC2 health check indicating it has been terminated or stopped"

For instructions on how to troubleshoot this issue, see An instance was taken out of service in response to an EC2 health check that indicated it had been terminated or stopped.

Troubleshoot instances that fail Amazon EC2 health checks

The following message indicates that your instance failed an Amazon EC2 status check:

"An instance was taken out of service in response to an EC2 instance status checks failure"

For instructions on how to troubleshoot this issue, see An instance was taken out of service in response to an Amazon EC2 instance status check failure.

Troubleshoot instances that fail ELB health checks

If you turned on ELB health checks for your group, then Amazon EC2 Auto Scaling performs both ELB and Amazon EC2 health checks. Amazon EC2 Auto Scaling terminates instances that fail either health check.

If your group has multiple attached target groups or load balancers, then all target groups and load balancers must pass health checks. If any target group or load balancer in your group fails a health check, then Amazon EC2 Auto Scaling terminates your instance.

The following message indicates that your instance failed an ELB health check:

"An instance was taken out of service in response to a ELB system health check failure"

For instructions on how to troubleshoot this issue, see An instance was taken out of service in response to an Elastic Load Balancing system health check failure.

Troubleshoot terminated EC2 Spot Instances

Amazon EC2 Auto Scaling terminates EC2 Spot Instances when capacity isn't available, or Spot price exceeds the maximum price that you specified for your instances.

Messages similar to the following example indicate that Amazon EC2 Auto Scaling terminated your instance:

"An instance was taken out of service in response to an EC2 Spot Instance interruption notice."

To prevent this issue from reoccurring, take one of the following actions to increase the capacity of your instance:

• Modify your allocation strategy to capacity-optimized.
• Increase the maximum price that you specify for your instance.
• Consider switching to on demand instances if your workload requires no interruptions. If you do, then modify your group configuration to launch On-Demand Instances.
• Use the advisor tool at the bottom of Amazon EC2 Spot Instances to identify instance types that have a lower rate of spot interruption. Then, modify your group to use those instance types.
  Note: If you must launch Spot Instances after you configure your launch template to use Spot Instances, then create a new launch template version. Then, update your group to use the new launch template.
  If you must launch spot instances with a mixed instances group, then update your mixed instance group to use the new instance types.

Troubleshoot instances that users terminate

Messages that resemble the following example indicate that a user or service terminates your instance with the TerminateInstanceInAutoScalingGroup API operation:

"An instance was taken out of service in response to a user request"

To prevent this issue from reoccurring, complete the following steps:

1. Open the AWS CloudTrail console.
2. In the navigation pane, choose Event history.
3. Check the history for TerminateInstanceInAutoScalingGroup API calls.
4. Review the API operations for the user or service that is made the call, then investigate why the user or service terminated the instance.

Troubleshoot "A user request update of AutoScalingGroup constraints" error messages

If you manually update your group's constraint, then Auto Scaling might terminate instances in that group to match the updated settings.

Messages that resemble the following example indicate that Amazon EC2 Auto Scaling terminated your instance because you updated a group's constraint:

"A user request update of AutoScalingGroup constraints to min: 0, max: 2, desired: 1 changing the desired capacity from 2 to 1"

To prevent this issue from reoccurring, complete the following steps:

1. Open the CloudTrail console.
2. In the navigation pane, choose Event history.
3. Check the history for UpdateAutoScalingGroup and SetDesiredCapacity API calls, then investigate why the user made the call that updated a group's constraint.

Troubleshoot instances that Amazon EC2 Auto Scaling terminates to balance instances with other zones

By default, Amazon EC2 Auto Scaling balances instances across all Availability Zones. When you add a new Availability Zone to a group, Amazon EC2 Auto Scaling launches a new instance in that zone. When Amazon EC2 Auto Scaling rebalances your instances, it might terminate instances in other zones. Amazon EC2 Auto Scaling also rebalances instances across Availability Zones when it can't launch an instance in an Availability Zone that you specify.

Messages that resemble the following example indicate that Amazon EC2 Auto Scaling terminated your instance to re-balance instances across Availability Zones:

"Instances were launched to balance instances in zones us-east-1a with other zones"

To prevent this error from reoccurring, suspend the Availability Zone Rebalance process for the group that Amazon EC2Auto Scaling terminated.

Important: It's a best practice to not suspend the AZ Rebalance process. If you suspend the process, then one Availability Zone might have more capacity than another. As a result the availability of that Availability Zone decreases.

If you must suspend the process, then complete the following steps:

1. Open the AWS Management Console.
2. From the navigation pane, choose Auto Scaling Groups.
3. Select the check box next to the group that Amazon EC2 Auto Scaling terminated. When you do, a split pane opens up in the bottom of the page.
4. On the Details tab, choose Advanced configurations, and then choose Edit.
5. For Suspended processes, select AZ Rebalance.
6. Choose Update.

Troubleshoot instances that Amazon EC2 Auto Scaling terminates when you update your purchase options

When you request an updated purchase option distribution, Amazon EC2 Auto Scaling rebalances your group.

Messages that resemble the following example indicate that Amazon EC2 Auto Scaling terminated On-Demand or Spot Instances to match your updated purchase option:

"An instance was taken out of service to balance the distribution of On-Demand and Spot capacities."

Important: It's a best practice to review the configuration of your new purchase option distribution and modify it to prevent this issue from recocurring.

Troubleshoot monitor alarms that trigger policy scale-in events

Messages similar to the following example indicate that Amazon EC2 Auto Scaling terminated your instance in response to an Amazon CloudWatch alarm that you configured:

"A monitor alarm triggered a policy scaledown changing the desired capacity"

To avoid this error reoccurring, modify either or both your automatic scaling policy and the CloudWatch alarm that activated it.

To modify the group policy that changed your group's capacity, complete the following steps

1. Open the AWS Management Console.
2. From the navigation pane, choose Auto Scaling Groups.
3. Select the check box next to your group
4. Choose the Automatic Scaling tab to view and edit policies.
5. Modify the policy that your CloudWatch alarm activated.

To modify your CloudWatch alarm, complete the following steps:

1. Open the CloudWatch console.
2. In the navigation pane, choose Alarms.
3. Choose the alarm, and then choose the History tab.
4. Check your history for any state changes to your alarm or modifications to the configuration of your alarm.
5. If the alarm state changed, then check the metric history at the time of the state change. Verify whether a metric value changed to greater or less than a threshold value that you specify in that time frame. If a metric changed, then investigate what changed the metric value.

Troubleshoot instances that fail custom checks.

When an instance fails a custom health check, Amazon EC2 Auto Scaling calls the SetInstanceHealth API operation and then sets the instance's state to Unhealthy. Amazon EC2 Auto Scaling terminates the unhealthy instance on its next run.

The messages similar to the following example indicate that Amazon EC2 Auto Scaling terminated your instance because it failed a custom health check:

"An instance was taken out of service in response to a user health-check"

To identify the user that created the custom health check that terminated your instance, review your event history for SetInstanceHealth API calls. Look for the user that ran the SetInstanceHealth API operation before the health check that your instance failed.

Troubleshoot scheduled action updates that change the constraints of your group

If a scheduled action decreases your group's capacity, then Amazon EC2 Auto Scaling terminates one or more instances to match the updated capacity.

The messages similar to the following example indicate that a scheduled action decreased your group's constraints:

"A scheduled action update of AutoScalingGroup constraints changed the desired capacity"

To prevent this issue from reoccurring, suspend the Scheduled Actions process that decreased your group's constraints. When you do, check your Scheduled actions for reocurring actions that terminate your instances, and either delete them or specify a new schedule for them. After you modify your Scheduled actions, be sure to reactivate the Scheduled Actions process.

Related information

Why did my Auto Scaling group scale in?

Why didn't Amazon EC2 Auto Scaling stop an unhealthy instance?