How do I troubleshoot Auto Scaling issues in Elastic Beanstalk?

7 minute read

I want to troubleshoot Amazon Elastic Compute Cloud (Amazon EC2) Auto Scaling issues in my AWS Elastic Beanstalk environment.

Short description

Auto Scaling issues in a load-balanced Elastic Beanstalk environment can include unexpected instance replacements, launch failures, or incorrect scaling behavior.

Note: Set capacity, scaling settings, and scheduled actions from your environment's Configuration page, the EB CLI, or option_settings in a configuration file. If you edit the Auto Scaling group directly in the Amazon EC2 console, then Elastic Beanstalk might overwrite your changes during the next environment update. Also, a single-instance environment uses Amazon EC2 Auto Scaling, but its minimum, maximum, and desired capacity are all fixed at 1. If you need scaling, then use a load-balanced, scalable environment. For more information, see Environment types.

Resolution

Check why the Auto Scaling group continuously replaces instances

Identify why your Amazon EC2 Auto Scaling group unexpectedly terminated your instances. If the Auto Scaling group terminated an unhealthy instance, then check the instance health check status. To resolve health check failures, see Troubleshoot unhealthy instances in Amazon EC2 Auto Scaling.

If an instance fails the Elastic Load Balancing or Amazon EC2 health check, then Amazon EC2 Auto Scaling marks the instance as unhealthy. Then, the Auto Scaling group terminates the instance and launches a replacement. If replacements repeat in a loop, then the new instances fail the same check. Confirm that your application starts, the health check path and port are correct, and the instances aren't overloaded.

Download the Elastic Beanstalk application logs to find the reason for the health check issues.

Check the following log files for deployment failures on Linux instances:

/var/log/eb-engine.log
/var/log/cfn-init-cmd.log
/var/log/cfn-init.log

Check the following web server logs for application traffic issues on Linux instances:

/var/log/nginx/access.log
/var/log/nginx/error.log
/var/log/httpd/access_log for Apache configurations only
/var/log/httpd/error_log for Apache configurations only

Check the following log files on Windows instances:

C:\cfn\log\cfn-init.log for CloudFormation initialization failures
C:\Program Files\Amazon\ElasticBeanstalk\logs\ for Elastic Beanstalk deployment failures
C:\inetpub\logs\LogFiles\W3SVC1\ for IIS access logs
C:\Windows\System32\LogFiles\HTTPERR\ for HTTP error logs
Windows Event Viewer console for Application, System, and Security logs. For more information, see Event Viewer on the Microsoft Learn website.

Resolve instance launch failures

If the Auto Scaling group can't launch instances, then the environment health transitions to Degraded or Severe. Check your environment's Events page and the Auto Scaling group's Activity history to find the launch error.

If you receive the "InsufficientInstanceCapacity" or "UnfulfillableCapacity" error, then AWS doesn't have enough capacity for the requested instance type in the Availability Zone. Retry after a few minutes, configure your environment to use more Availability Zones, or choose a different instance type. For more information, see How do I resolve "InsufficientInstanceCapacity" or "UnfulfillableCapacity" launch errors in my Amazon EC2 Auto Scaling group?

If you receive the "VcpuLimitExceeded" or "InstanceLimitExceeded" error, then you reached an Amazon EC2 service quota for the instance family. Request a quota increase, or use an instance type within your quota. For more information, see How do I troubleshoot InstanceLimitExceeded errors when starting or launching an EC2 instance? and Amazon EC2 service quotas.

If the launch configuration is not valid, then the instance type isn't available in the selected Availability Zone. The AMI, key pair, subnet, or security group might not be valid. Update the environment capacity or instance settings in Elastic Beanstalk.

If instances can't launch for a reason related to your account, then contact AWS Support.

Check why the Auto Scaling group doesn't scale out before 100% CPU utilization

If the CloudWatch alarm associated with your scaling policy uses Average statistics, then the Auto Scaling group might not respond to a sudden traffic increase. The CPU utilization maximum value can reach 100% even though the average stays below your threshold.

To resolve this issue, change the Statistic setting from Average to Maximum.

Verify the CloudWatch alarm metric matches your load balancer type

If the Auto Scaling group doesn't scale out or in as expected, then verify the associated CloudWatch alarm for misconfigured metrics. For example, the Latency metric belongs to Classic Load Balancer. If you use an Application Load Balancer, then use TargetResponseTime instead.

Check why the environment doesn't scale in

If the instance count doesn't decrease after load returns to normal, then take the following actions:

Review the scale-down settings. The Lower threshold, Metric, Statistic, Period, and Breach duration determine when a scale-in operation runs. If the metric doesn't fall below the lower threshold for the full breach duration, then no scale-in occurs. Review these settings on the Configuration page under Capacity. For more information, see Auto Scaling triggers for your Elastic Beanstalk environment.
Check the cooldown period. A long cooldown delays consecutive scaling actions. For more information, see Scaling cooldowns for Amazon EC2 Auto Scaling.
Check for scheduled actions. A scheduled action might have raised the minimum or desired capacity. For more information, see Scheduled Auto Scaling actions for your Elastic Beanstalk environments.
Confirm the minimum instance count. The group never scales below the minimum instance count that you set. Verify that the minimum is the value you expect.

Check why the environment runs a different number of instances than configured

If the running instance count doesn't match your capacity settings, then take the following actions:

Check the environment type. In a single-instance environment, the minimum, maximum, and desired capacity are all set to 1. To scale, change to a load-balanced environment. For more information, see Environment types.
Verify where you set capacity. Set the minimum and maximum instances from your environment's Configuration page under Capacity, the EB CLI, or the aws:autoscaling:asg namespace in a configuration file. Changes that you make directly in the Amazon EC2 console might be overwritten. For more information, see Auto Scaling your Elastic Beanstalk environment instances.
Check for scheduled actions. A scheduled action can change the minimum, maximum, or desired capacity. After a scheduled action ends, Amazon EC2 Auto Scaling doesn't return to the previous settings automatically. For more information, see Scheduled Auto Scaling actions for your Elastic Beanstalk environments.
Check for updates and rebuilds. A managed platform update, deployment, or environment rebuild can temporarily change the instance count during the replacement process.
Check for immutable updates. During an immutable environment update, the capacity doubles temporarily. Instances in the new Auto Scaling group start to serve requests before Amazon EC2 Auto Scaling terminates the original group's instances. For more information, see Immutable environment updates.

Reduce frequent and unnecessary scaling events

Narrow threshold configurations create a small window between scale-up and scale-down, which causes reactive scaling cycles.

To resolve this issue, take the following actions:

Increase the gap between the Upper threshold and Lower threshold settings of the CloudWatch alarms.
Implement cooldown periods.
Perform load testing, and then use historical CloudWatch metrics to fine-tune these settings.

Configure scaling on a custom metric or a schedule

By default, Elastic Beanstalk doesn't collect all metrics. To scale on a custom metric, first publish the metric to CloudWatch from your instances with the CloudWatch agent. Then, on the Configuration page, in the Capacity category, choose Edit, and then set the scaling metric to your custom metric. For more information, see Auto Scaling triggers for your Elastic Beanstalk environment.

If your application has predictable periods of peak traffic, then use time-based scaling. On the Configuration page, in the Capacity category, under Time-based scaling, add a scheduled action with a minimum, maximum, and optional desired capacity. Add a second scheduled action to return to your baseline capacity. For more information, see Scheduled Auto Scaling actions for your Elastic Beanstalk environments.