How do I resolve throttling errors in my CloudWatch logs?

Short description

If you exceed the API rate limit in CloudWatch logs, then you might receive a RequestLimitExceeded or ThrottlingException error due to API throttling. Resolve your throttling errors based on the following scenarios:

• Intermittent throttling on CloudWatch logs occurs when you access the logs.
• ThrottlingException errors occur when you use an application or script to retrieve CloudWatch log data.
• Throttling errors occur when you integrate PutLogEvents APIs with an AWS Lambda function.
• Throttling errors occur for PutResourcePolicy:ResourceLimitExceeded when you turn on logs for other AWS services.

Resolution

Intermittent throttling on CloudWatch logs occurs when you access the logs

The FilterLogEvents and GetLogEvents APIs are used to list your log events from a specific log stream or a log group. These APIs have a hard limit and don't qualify for a limit increase. If you use these APIs to search for log events and you reach the hard limit, then a RateExceeded error appears. For more information on hard limits, see CloudWatch Logs quotas.

The following are best practices to prevent throttling errors in this scenario:

• Use a subscription filter to retrieve log data from CloudWatch logs in real time. For more information, see Log group-level subscription filters and Real-time processing of log data with subscriptions.
• Use CloudWatch Logs Insights to retrieve log data from CloudWatch logs. Also, use queries to filter your logs to view specific log groups.
• Export log data to Amazon Simple Storage Service (Amazon S3) for batch use cases. Don't use this method for real time analysis and processing. Log data might take up to 12 hours to become available for export from CloudWatch Logs.

ThrottlingException errors occur when you use an application or script to retrieve CloudWatch log data

Collector scripts are used to collect CloudWatch logs and uses a DescribeLogStream or GetLogEvents API call to pull the data. Data is pulled from different log streams or different time frames in the same log group. You might receive ThrottlingException errors when you use these APIs.

The following are best practices to prevent throttling errors in this scenario:

• Use exponential backoff and retries when you make an API call. For more information, see Exponential backoff and jitter and Retry behavior.
• Use scheduled actions with randomization to distribute your APIs over a period of time.
• Add sleep intervals between consecutive API calls. Also, add delays between APIs that are sent from the same script or application. If you send APIs in rapid succession, then rate errors are more likely to occur.
• Use a SIEM solution such as Splunk to retrieve logs from CloudWatch. SIEM solutions are used to gather data from multiple systems and analyzes the data to detect unusual behavior. You might experience API throttling when you use the Splunk plugin. To prevent this issue, create a CloudWatch logs subscription filter with Amazon Kinesis Data Firehose and deliver the log data to Splunk. For more information, see Configure Kinesis inputs for the Splunk Add-on for AWS on the Splunk website.

Throttling errors occur when you integrate PutLogEvents API calls with an AWS Lambda function

The PutLogEvents API is used to upload logs to a specified log stream in batches of one MB. This API has a default rate limit of 2500 TPS per account per AWS Region. The following Regions have a rate limit of 5000 TPS:

• US East (N. Virginia)
• US West (Oregon)
• Europe (Ireland)
• Europe (Milan)
• Europe (Paris)
• Europe (Stockholm)

To resolve TPS throttling errors for this scenario, request a quota increase.

Throttling errors occur for PutResourcePolicy:ResourceLimitExceeded when you turn on logs for other AWS services

The PutResourcePolicy API is used to create or update a policy to allow other AWS services to place log events into a specific account. Accounts can have a maximum of 10 resource policies per Region. This is a hard limit and can't be increased.

To list the policies that you have in a Region, run the following command:

Note: Replace example-region with the required Region.

$ aws logs describe-resource-policies --region example-region

To resolve this throttling error, use the delete-resource-policy command to delete any resource policies that aren't required or not used.

Another method is to combine multiple policy statements into a single policy. Use this method to make sure that the number of resource policies doesn't exceed the limit of 10.

Related information

How do I avoid throttling when I call PutMetricData in the CloudWatch API?