Performance degradation after 500 IOPS using AWS DynamoDB java SDK 2

I'm assuming you have IOPS scaling setup for the table and also for indexes. If auto-scaling is not set or if the max IOPS lower than your consumption, then you will see your queries throttling.

When working with high throughput operations in DynamoDB using the AWS Java SDK 2, there are several factors that could be causing performance degradation beyond 500 parallel calls.

First, it's important to understand that DynamoDB applies rate limits per second, even though CloudWatch only reports minute-level metrics. This means you might be experiencing throttling that isn't immediately visible in the console. If too many requests occur within a small timeframe (like a few seconds), subsequent requests can be throttled even if the average throughput for the minute appears below your provisioned capacity.

For your specific issue with Java virtual threads, consider these optimization strategies:

1. Connection Management: The SDK has limits on concurrent connections. While DAX clients can handle up to 40,000 concurrent connections, having too many connections from a single client instance can overwhelm the service. Start with lower concurrent connection settings and gradually increase based on performance testing.

2. Implement Exponential Backoff and Jitter: Add jitter and exponential backoff to your API calls to spread out the request load and handle any throttling more gracefully.

3. Check for Hot Keys: Even with UUIDs, you might be experiencing a "rolling hot key" scenario where requests target specific partitions sequentially. This can cause throttling at the partition level (3000 RCU or 1000 WCU per partition) even when your overall table capacity isn't exceeded.

4. Optimize Batch Operations: If you're using batch operations, remember there's a limit of 100 items per request. Consider implementing pagination to process data in smaller chunks.

5. Client-Side Optimization: Implement a singleton client instance pattern to reuse connections rather than creating new ones for each request.

6. Consider On-Demand Mode: If your workload is spiky, on-demand capacity mode might handle the load better, though you should be aware it also has maximum throughput settings.

For your Java virtual threads implementation specifically, consider that while virtual threads are lightweight, the underlying network I/O operations still require system resources. The performance degradation might be related to resource contention at the OS or network level rather than DynamoDB itself.
Sources
DynamoDB AWS SDK SdkIterable taking time to iterate | AWS re:Post
Troubleshooting throttling issues for provisioned mode - Amazon DynamoDB
Configuring your DAX client - Amazon DynamoDB
How to create items in DynamoDB without throttling? | AWS re:Post
DynamoDB maximum throughput for on-demand tables - Amazon DynamoDB