How do I troubleshoot auto scaling issues in Amazon ECS?

Short description

To automatically update the desired task count, set up AWS Application Auto Scaling and Amazon CloudWatch alarms for Amazon ECS.

Application Auto Scaling might not add or remove tasks as expected because of the following reasons:

• You didn't correctly configure the scaling policies.
• You deleted or edited the CloudWatch alarm that invokes the scaling policies.
• The auto scaling is suspended.
• You incorrectly specified the cron expression format in the scheduled action.
• You used AWS CloudFormation or AWS Cloud Development Kit (AWS CDK) to manually update the desired task count and entered the wrong value.
• The desired number of tasks is incorrectly configured.
• Your Amazon ECS cluster doesn't have enough resources or capacity to run new tasks.
• You're experiencing network issues.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Troubleshoot CloudWatch alarms and scaling policies

To troubleshoot CloudWatch alarms and scaling policies, complete the following tasks based on the scaling policy that you use.

Scalable target

To check whether the service is registered as a scalable target with Application Auto Scaling, run the describe-scalable-targets command:

aws application-autoscaling describe-scalable-targets --service-namespace ecs --region example-region

Note: Replace example-region with your AWS Region.

If the service isn't registered, then run the register-scalable-target command to register the service:

aws application-autoscaling register-scalable-target --service-namespace ecs --scalable-dimension ecs:service:DesiredCount --resource-id service/your-cluster/your-service-name --min-capacity 1 --max-capacity 10 --region example-region

Note: Replace your-cluster with your Amazon ECS cluster name, your-service-name with your service name, and example-region with your Region.

In the preceding command's output, check the value of DynamicScalingInSuspended, DynamicScalingOutSuspended, and ScheduledScalingSuspended. When DynamicScalingInSuspended or DynamicScalingOutSuspended is set to true, Application Auto Scaling doesn't update capacity when you initiate a scaling policy. When ScheduledScalingSuspended is set to true, Application Auto Scaling doesn't initiate the scaling actions that you scheduled to run during the suspension period. For more information about how to resume suspended auto scaling, see Suspend and resume scaling for Application Auto Scaling.

After you register the service, configure the scaling policies and CloudWatch alarms.

Step scaling

Check whether CloudWatch is initiating the alarms that are associated with the scaling policies. You can find errors in the CloudWatch alarm's history. If there's no step adjustment for the breach delta, then you see the following error: "Failed to execute AutoScaling action: No step adjustment found for metric value [xx, xx] and breach delta xx". To resolve this issue, make sure that you set the threshold, step adjustments, and scaling adjustment type in the CloudWatch alarm. Also, specify all step adjustments in your policies. For scale in, enter -infinity to 0, and for scale out, enter 0 to +infinity.

Note: If you use Amazon ECS capacity providers, then make sure that there are enough container instances for the tasks to run on.

Target tracking scaling

Because target tracking scaling policies can affect scaling behavior, don't edit or delete associated CloudWatch alarms. If you modify or delete these alarms, then you must recreate the target tracking policy.

When you configure multiple scaling policies, the policies can conflict and cause consecutive scale-in and scale-out activity. Each policy must use a different metric if you want to have multiple target tracking scaling policies for an Amazon ECS service.

Troubleshoot incorrect cron expression

Make sure that the cron expression in the scheduled actions configuration for Application Auto Scaling uses the following format: [Minutes] [Hours] [Day_of_Month] [Month] [Day_of_Week] [Year]. The six fields must be separated by a blank space.

For an example action that uses the cron expression, see Create a scheduled action that runs on a recurring schedule.

Troubleshoot the desired task count update

When you update the desired task count for an Amazon ECS service, you might invoke a scale event. For scale out, Application Auto Scaling increases the desired count to the minimum capacity. For scale in, Application Auto Scaling increases the desired count to the maximum capacity. In both cases, Application Auto Scaling continues to scale based on the scaling policy, but doesn't modify the desired count.

If you create your Amazon ECS service with CloudFormation or AWS CDK, then the default DesiredCount value is 1. If you update your Amazon ECS service with CloudFormation or AWS CDK, then the application uses the existing DesiredCount for new deployments.

For the desired count value in the CloudFormation stack or AWS CDK, specify a value between the minimum and maximum values during the service update.

Note: Make sure that your desired count meets the needs of the service's deployment configuration, particularly the minimum and maximum percent settings.

Troubleshoot cluster capacity issues

To avoid capacity issues, use Amazon ECS capacity providers to automatically provision Amazon Elastic Compute Cloud (Amazon EC2) instances.

To prevent excessive scaling, auto scaling uses the actual running task count in a service as the starting point, not the desired count. If there aren't enough container instance resources to place the additional tasks, then auto scaling can't complete scaling. To resolve this issue, use capacity providers to automatically scale the container instance capacity. Otherwise, you must manually increase the container instance capacity.

After the cooldown period, auto scaling continues scaling activities. For AWS Fargate tasks, use Fargate capacity providers to manage compute capacity.

Auto scaling uses the actual running task count, not the desired count, as the starting point for scaling. If there are insufficient container instance resources to place the additional tasks, then auto scaling can't complete the scaling activity. However, to prevent excessive scaling attempts that can't be fulfilled, auto scaling still tries to scale as much as possible based on the available resources.

Troubleshoot network-related scaling issues

Check the configuration of tasks in awsvpc network mode

When auto scaling increases the number of tasks, each task in awsvpc network mode requires its own IP address. If your virtual private cloud (VPC) subnets don't have enough available IP addresses, then the number of tasks don't scale up. Regularly monitor your VPC's IP address use. If your configuration is near the capacity limits, then use larger CIDR blocks or create new subnets.

Amazon ECS tasks in awsvpc network mode use elastic network interfaces. There's a maximum quota for total network interfaces in each instance type in each VPC. If you exceed this quota, then your tasks don't launch. To check your network interface use, complete the following steps:

1. Open the Amazon EC2 console.
2. From the navigation pane, choose Network interfaces.
3. Use the Instance ID column to view the instances that each network interface is attached to. To view more information about the network interface such as IP addresses and security groups, choose the network interface.

Verify that your configuration doesn't block required traffic

Security groups or network access control list (network ACL) rules can block required traffic and cause health check failures that lead to unnecessary scaling events. Make sure that your security groups allow inbound traffic to your application's ports. Also make sure that your network ACLs allow inbound and outbound traffic between your application and Amazon ECS.

Related information

Automatically scale your Amazon ECS service

application-autoscaling