AWS Glue: DDB write_dynamic_frame_from_options fails with requests throttled

Question

Hi,
I have aws glue job written in Python that reads DDB table (cross accounts) and then attempts to write to another table in a current account.
The code for writing DDB is very simple
```
def WriteTable(gc, dyf, tableName):
    gc.write_dynamic_frame_from_options(
        frame=dyf,
        connection_type="dynamodb",
        connection_options={
            "dynamodb.output.tableName": tableName,
            "dynamodb.throughput.write.percent": "1"
        }
    ) 
```

I run it on a small test table with 20 mil items and 500MB in size.

Avg item size is 28 bytes and below is the item sample.

The table has primary key as a `pair` + sort key `date`

```
{
 "pair": {
  "S": "AMDANG"
 },
 "date": {
  "N": "20080101"
 },
 "value": {
  "N": "0.005845"
 }
}
```

Read sink works just fine. Job reads items pretty fast with 3-5 workers and it takes a few minutes.

However, the write sink is a disaster. The requests get throttled and it barely can write 100k records a minute.
I tried already all Glue versions and 3-10 workers.

It actually fails fast with retries exhausted.
I had to change "dynamodb.output.retry" to 30-50 because default 10 just fails glue job as soon as it starts writing with: An error occurred while calling o70.pyWriteDynamicFrame. DynamoDB write exceeds max retry 10

[ddb metrics screenshot when glue writes](https://user-images.githubusercontent.com/62673610/144349675-431ce14c-e92d-4f9b-a78b-db71d15b2d3d.png)
Lots of Write Throttle events.

What could be the problem here? I thought it could be a primary key distribution problem, but they should be well distributed.
I've looked at the `pair` distinct count, there are a few thousand pairs. The dates are daily sequences for multiple years.

Ref: https://github.com/awslabs/aws-glue-libs/issues/104

Answer

The issue is that DynamoDB cannot auto-scale fast enough to keep up with the AWS Glue write speed. To avoid DynamoDB ThrottlingException on write please use Capacity mode “Provisioned” with Autoscaling for Read and Write. Choose the min / max capacity wisely. Set the target utilization to allow DynamoDB sufficient time to scale. As an example I ran this blog successfully setting the number of workers in the Glue job to 25 and Provisioned Table capacity with autoscaling for DDB for set to Read 5 (min) 40K (max) TU 70% Write 5K / 20K (min) 40K (max) TU 35% Total records / items : 14,092,383

Essentially providing enough time for DynamoDB to auto scale.

Kunal_G

answered 2 years ago

EXPERT

John_F

reviewed 2 years ago

Essentially providing enough time for DynamoDB to auto scale.

Answer

While using on-demand table is recommended, it too must scale to provide your throughput needs. DynamoDB on-demand tables provide 4k WCU out of the box, so your ETL job will be initially capped at 4k WCU, then throttle until DynamoDB makes the necessary changes to accommodate your request throughput. To achieve best results, you must first "pre-warm" your table, this is easier done on new tables, but will also work for old ones.

Create table in provisioned mode with no auto-scaling, assign 40k WCU and 40 RCU for capacity settings.
Allow table to go from Creating/Updating to Activce
Once Active, change capacity mode to on-demand.
You can now achieve approx 40k WCU out of the box, and your ETL job will have the capacity it needs to run with no issue.

*note that 40k WCU is the soft limit for max table capacity, and can be increased with a request from Service Quotas.

1. Create table in provisioned mode with no auto-scaling, assign 40k WCU and 40 RCU for capacity settings. 
2. Allow table to go from Creating/Updating to Activce
3. Once Active, change capacity mode to on-demand. 
4. You can now achieve approx 40k WCU out of the box, and your ETL job will have the capacity it needs to run with no issue.

*note that 40k WCU is the soft limit for max table capacity, and can be increased with a request from [Service Quotas](https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html).

AWS Glue: DDB write_dynamic_frame_from_options fails with requests throttled

Relevant content