I want to troubleshoot "Rate Exceeded" errors when the application calls to Amazon EC2 Auto Scaling, AWS Auto Scaling, or AWS Application Auto Scaling.
Short description
To resolve "Rate Exceeded" errors and throttling, check that your application makes only necessary calls. If you still receive "Rate Exceeded" errors, then request an AWS service quota increase.
If API requests exceed the maximum rate, then you receive a "Rate Exceeded" error, and the service throttles further API calls.
Resolution
Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.
Review the following limitations:
- API calls can't exceed the maximum allowed API request rate per AWS account and per AWS Region. The maximum includes API calls from the AWS CLI and the AWS Management Console.
- Amazon EC2 Auto Scaling, AWS Auto Scaling, and Application Auto Scaling have their own API throttle buckets. All Amazon EC2 Auto Scaling API calls have a single, shared API limit. Amazon EC2 Auto Scaling API calls don't affect the limit for Application Auto Scaling APIs.
For more information, see the Auto Scaling group API operations section of Quotas for Auto Scaling Resources and groups.
Check that your application makes only necessary calls
Check current Auto Scaling API Usage
To check current Auto Scaling API usage, monitor the AWS Usage statistics in your Amazon CloudWatch metrics for the Amazon EC2 Auto Scaling service. Use this information to optimize and reduce your API usage.
Validate "describe" calls
Excessive describe calls contribute to the total API requests that are measured against the allowed request rate. Verify that your application calls only necessary describe operations. For example, use push notifications from Amazon EventBridge to send notifications when Amazon Elastic Compute Cloud (Amazon EC2) instances change state, such as when they start a lifecycle hook.
Applications that check the lifecycle state of the Amazon EC2 instance don't call describe operations. Instead, these applications query EC2 instance metadata. You must retrieve tag information, including the name of the Auto Scaling group, from instance metadata. Use the launch template to turn on tag information for the instance.
Check calls from third-party applications
Review your third-party applications for unnecessary API operations. If third-party applications call unnecessary API operations, then update the settings of your third-party applications.
Use error retries and exponential backoffs
Error retries and exponential backoffs limit the rate of API calls. Each AWS SDK implements automatic retry logic and exponential backoff algorithms. For more information, see Retry behavior.
Avoid bursts of activity
Avoid situations that call API operations in bursts. For example, When you launch instances in an Amazon EC2 Auto Scaling group, don't activate scale-in protection for individual instances. Turn on the option by default for the Auto Scaling group so that all instances have protection.
Request a service quota increase
Use the AWS CLI or the AWS Management Console to request a quota increase.
If the Service Quota Increase dashboard is unavailable, then request an increase from the Support Center console. Under the service quota Increase section, complete the following steps:
- For Limit type, choose Auto Scaling.
- For Auto Scaling Service, choose EC2 Auto Scaling.
- For Limit, choose API throttling.
- For New limit value, choose N/A.
Include the following information in the request:
- Validation of your API call rate and the implementation of error retires or exponential backoff methods
- AWS Region and timeframe related to the throttling issues
- Full workflow of the AutoScaling APIs
- When you expect your peak API usage to happen, such as during a deployment
For more information, see How do I manage my AWS service quotas?