Why is my Amazon DynamoDB table being throttled?

Short description

Here are some of the common throttling issues that you might face:

• Your DynamoDB table has adequate provisioned capacity, but most of the requests are being throttled.
• You activated AWS Application Auto Scaling for DynamoDB, but your DynamoDB table is being throttled.
• Your DynamoDB table is in on-demand capacity mode, but the table is being throttled.
• You have a hot partition in your table.
• Your table's traffic is exceeding your account throughput quotas.

Resolution

For information on DynamoDB metrics that must be monitored during throttling events, see DynamoDB metrics and dimensions. These metrics might help you to locate the operations creating throttled requests and identify the cause for throttling. Use one or more of the following troubleshooting options based on your use case.

Your DynamoDB table has adequate provisioned capacity, but most of the requests are being throttled

DynamoDB reports minute-level metrics to Amazon CloudWatch. The metrics are calculated as the sum for a minute and then averaged. However, the DynamoDB rate limits are applied per second. For example, if you provisioned 60 write capacity units for your DynamoDB table, then you can perform 3600 writes in one minute. However, driving all 3600 requests in one second with no requests for the rest of that minute might result in throttling. The total number of read capacity units or write capacity units per minute might be lower than the provisioned throughput for the table. However, if all the workload falls within a couple of seconds, then the requests might be throttled.

To resolve this issue, make sure that your table has enough capacity to serve your traffic and retry throttled requests using exponential backoff. If you are using the AWS SDK, then this logic is implemented by default. For more information, see Error retries and exponential backoff.

Note: DynamoDB doesn't necessarily start throttling the table after the consumed capacity per second exceeds the provisioned capacity. With the burst capacity feature, DynamoDB reserves a portion of the unused capacity for later bursts of throughput to handle usage spikes.

You activated AWS Application Auto Scaling, but your table is still being throttled

AWS Application Auto Scaling isn't a suitable solution to address sudden spikes in traffic with DynamoDB tables. Application Auto Scaling initiates a scale up only when two consecutive data points for consumed capacity units exceed the configured target utilization value within a one-minute span. Application Auto Scaling automatically scales the provisioned capacity only when the consumed capacity is higher than target utilization for two consistent minutes. Also, a scale-down event is initiated when 15 consecutive data points for consumed capacity in CloudWatch are lower than the target utilization. After Application Auto Scaling is initiated, an UpdateTable API call is invoked that might take a couple of minutes to update the provisioned capacity for your DynamoDB table or index. Application Auto Scaling requires consecutive data points with higher target utilization values to scale up the provisioned capacity of the DynamoDB table. During this period, any requests that exceed the provisioned capacity of the table are throttled. Therefore, it's not a best practice to use Application Auto Scaling to handle spiky workloads in DynamoDB and you might consider switching to on-demand mode in that use case. For more information, see Managing throughput capacity automatically with DynamoDB auto scaling.

Your table uses the on-demand capacity mode, but the table is still being throttled

When the table uses the on-demand capacity mode, the table doesn't throttle as long as the following conditions are true:

• The access pattern is distributed evenly across partitions to avoid issues related to hot partition.
• The table doesn't exceed double its previous peak traffic.

For on-demand tables, DynamoDB automatically allocates more capacity as your traffic volume increases to make sure that your workload doesn't experience throttling. However, throttling can occur if the traffic volume is more than double the previous peak within a span of 30 minutes. For more information, see Peak traffic and scaling properties.

You have a hot partition in your table

In DynamoDB, a partition key that doesn't have a high cardinality can result in many requests targeting only a few partitions and resulting in a hot partition. A hot partition can cause throttling if the partition limits of 3000 RCU or 1000 WCU (or a combination of both) per second are exceeded.

To find the most accessed and throttled items in your table, use the Amazon CloudWatch Contributor Insights. Amazon CloudWatch Contributor Insights is a diagnostic tool that provides a summarized view of your DynamoDB tables traffic trends and helps you identify the most frequently accessed partition keys. With this tool, you can continuously monitor the graphs for your table’s item access patterns. A hot partition can degrade the overall performance of your table. To avoid this poor performance, distribute the read and write operations as evenly as possible across your table. For more information, see Designing partition keys to distribute your workload evenly and Choosing the right DynamoDB partition key.

Note: Using the CloudWatch Contributor Insights tool for DynamoDB incurs additional charges. For more information, see CloudWatch Contributor Insights for DynamoDB billing.

Your table's traffic is exceeding your account throughput quotas

The table-level read throughput and table-level write throughput quotas apply at the account level in any Region. These quotas apply for tables with both provisioned capacity mode and on-demand capacity mode. By default, the throughput quota placed on your table is 40,000 read requests units and 40,000 write requests units. If the traffic to your table exceeds this quota, then the table might be throttled. To resolve this issue, use the Service Quotas console to increase the table-level read or write throughput quota for your account.

Related information

Using write sharding to distribute workloads evenly