How do I resolve the ThrottlingException error in Timestream for LiveAnalytics?

Short description

You get the "ThrottlingException" error when the following requests exceed the quota within a specific time frame:

• Queries
• Parameter Store, a capability of AWS Systems Manager, API calls
• Active magnetic store writes

Resolution

Too many query requests

When you exceed the query request rate quota, you receive the following error message: 

"An error occurred (ThrottlingException) when calling the QueryOperation (reached max retries: 4): Request rate limit exceeded".

Note: The request rate quota is dynamic and depends on the overall resource usage at the service level.

If your queries scan more data than necessary and cause multiple paginated calls, then use measure names or customer-defined partitioning keys to filter out unnecessary data. For example, if your queries scan entire table data, then add time predicates so that the query engine removes unnecessary data scans. It's a best practice to configure frequently accessed key collections as partition keys for the table so that Timestream for LiveAnalytics can efficiently manage internal partitioning.

To optimize your underlying data model, see Data modeling. To improve performance and reduce latency, see Queries.

Too many Parameter Store API requests

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

When you exceed the Parameter Store API request quota for each AWS account and AWS Region, you receive one of the following error messages:

• "An error occurred (ThrottlingException) when calling the GetParameters operation (reached max retries: 4): Rate exceeded."
• "An error occurred (ThrottlingException) when calling the GetParametersByPath operation (reached max retries: 4): Rate exceeded."
• "An error occurred (ThrottlingException) when calling the DescribeParameters operation (reached max retries: 2): Rate exceeded."

To reduce the frequency of Parameter Store API calls, take the following actions:

• Schedule API calls at staggered intervals.
• Use APIs that can return multiple values in a single call. For example, GetParameters and GetParametersByPath can retrieve up to 10 parameters in one API call.
• Implement error retries and exponential backoff in your API calls. To prevent concurrent API calls, incorporate jitter into the retry logic. To more evenly distribute request spikes at a constant rate and reduce throttling incidents, add random intervals to the backoff.

You can modify your code to include a 1-second pause between calls. The AWS CLI includes retry settings that you can modify. For example you can adjust the setting to 50 milliseconds before the first retry, 100 milliseconds before the second, and 200 milliseconds before the third.

If you use AWS Lambda to make API calls to Parameter Store, then implement parameter caching.

To implement batch processing, divide large historical datasets into smaller, manageable batches. Process data in chunks according to write request quotas.

Too many active magnetic store write requests

When you exceed the quota of 250 active magnetic store write requests, you receive the following error message: 

"An error occurred (ThrottlingException) when calling the WriteRecords operation (reached max retries: 5): Your magnetic store writes to Timestream for LiveAnalytics are throttled for this database."

Note: When a partition receives data, the partition might remain in an active state for up to 6 hours after the ingestion process.

You can't increase the 250 quota.

If the ActiveMagneticStorePartitions metric exceeds 250, then take the following actions:

• Reduce the number of different data series that your application writes concurrently.
• Reduce the duration of time that the active magnetic partitions concurrently ingest data. For example, if the partitions ingest 24 hours of data in one operation, then break the data into smaller time chunks.
• Use Batching writes to reduce the potential throttling behavior.

Note: When there's a large amount of data, it's best practice to create batch load jobs.

If you still receive magnetic throttling errors and the dead-letter queue (DLQ) receives many records, then locate the tables that are experiencing the magnetic throttling. Use the memory store for high throughput ingestion workloads. For more information, see Handling WriteRecords throttles.