Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

WriteThrottleEvents despite sufficient warm throughput

0

Hi everyone,

We’re running a DynamoDB table in on-demand mode and experiencing unexpected WriteThrottleEvents.

Here’s the setup:

  • Write throughput: ~100,000 WPS (write operations per second)
  • Table has warmed up to support 200,000 WCU (we’ve steadily scaled traffic up)
  • Read throughput: ~60,000 RPS
  • Table is in on-demand mode
  • Approx. 5,000 WriteThrottleEvents per second still occurring
  • Write key is an email address (very high cardinality)
  • Writes are well-distributed across different partition keys — we assume no hot partitioning

Questions:

  • What could cause throttling despite warm capacity being above actual usage?
Topics
Database
Tags
Amazon DynamoDB
Language
English

asked a year ago454 views

3 Answers
1

Without more information its difficult for me to assist. If you are certain that no more than 5RPS per partition key is certain, then here is some things I would suggest looking at:

  1. GSI back-pressure, this is where you have a GSI on your table with a partition key which is low in cardinality. This causes throttling on the index, which pushes back on the table to ultimately throttle the client.
  2. Check your account/table level limits, ensure you are not exceeding your account limits defined in Service Quotas.
  3. If both of the above are not the issue, I suggest enabling CloudWatch Contributor Insights which can identify hot keys, and help you better understand the cause of throttling.
AWS
EXPERT

answered a year ago

  • Nicolas
    a year ago

    is it possible that the read warm throughput can be a bottleneck when each writes contain a ConditionalCheck ?

  • Leeroy Hannigan EXPERT
    a year ago

    No, conditional checks do not consume read capacity units.

0

DynamoDB does not guarantee unlimited per-partition throughput, even in On-Demand mode. On-Demand scaling handles overall capacity — but each partition still has an internal maximum. 1,000 WCU or 3,000 RCU per partition per second

If writes cluster momentarily to a few partitions, you get throttling even if the total traffic is low relative to your warmed capacity.

https://aws.amazon.com/blogs/database/demystifying-amazon-dynamodb-on-demand-capacity-mode/ [refer myth #10, On-demand tables will fix my throttling]

EXPERT

answered a year ago

  • Nicolas
    a year ago

    i understand, but since the actual number of write requests per item should be less < 5 and our PK is an email address i am not sure how this could be an issue of a hot partition ?

  • Nicolas
    a year ago

    the writes are spread across all keys of the table (e-mail addresses), with 2-5 writes per item

0

i have increased the warm read throughput to the same amount, and now it works

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content