How to create items in DynamoDB without throttling?

My assumption here is that you read the items to delete by using a Scan operation. When you do this, items are returned using the same sort order as they are stored in DynamoDB. So now when you call DeleteItem in order, you are essentially causing a "Rolling hot key", wherein, all of your deletes are targetted to one partition, and slowly moving through all partitions one at a time in sequential order.

Here is my advice to obtain better performance:

1. If you use Scan, either shuffle the data, or assign each one of your worker threads a different segment using Parallel Scan.
2. Pre-warm your table, with the above in place, and also pre-warming your table you will have the ability to scale effortlessly as your items are all unique.

import boto3
import threading
from concurrent.futures import ThreadPoolExecutor

dynamodb = boto3.resource('dynamodb')
TABLE_NAME = 'MyTable'
PRIMARY_KEY = 'PK'
table = dynamodb.Table(TABLE_NAME)

def scan_and_delete(segment, total_segments):
    try:
        scan_kwargs = {
            'Segment': segment,
            'TotalSegments': total_segments,
            'ProjectionExpression': PRIMARY_KEY
        }

        while True:
            response = table.scan(**scan_kwargs)
            items = response.get('Items', [])
            
            for item in items:
                key = {k: item[k] for k in item if k in [PRIMARY_KEY]}
                table.delete_item(Key=key)
                print(f"Deleted item: {key}")

            if 'LastEvaluatedKey' in response:
                scan_kwargs['ExclusiveStartKey'] = response['LastEvaluatedKey']
            else:
                break
    except Exception as e:
        print(f"Error in segment {segment}: {e}")

NUM_THREADS = 20

with ThreadPoolExecutor(max_workers=NUM_THREADS) as executor:
    for i in range(NUM_THREADS):
        executor.submit(scan_and_delete, i, NUM_THREADS)