High P99 Latency on DynamoDB via VPC Endpoint during Low TPS (Java CRT Client)

Based on the behavior you're describing, here are some insights that may help address your low-TPS latency issues:

Understanding the Latency Pattern: Some variation in latency metrics, particularly in higher percentiles like P99, is expected in DynamoDB and can result from background operations that maintain high availability and durability, or transient infrastructure issues. However, the consistent pattern you're seeing during low traffic suggests connection management issues rather than DynamoDB service-side problems.

Connection Reuse and Configuration: The AWS SDK for Java 2.x emphasizes the importance of reusing service clients to improve performance by reducing time spent establishing new connections. When clients are not reused effectively or connections expire during low-traffic periods, you'll see the "cold start" latency you're experiencing. For your specific questions:

1. Connection Settings: While there isn't a universally prescribed "sweet spot" for connectionMaxIdleTime or maxConcurrency specific to CRT, the key is balancing connection retention with resource efficiency. Consider setting connectionMaxIdleTime high enough to span your low-traffic periods (potentially several minutes) to prevent premature connection closure. However, be mindful that keeping too many idle connections can consume resources.

2. VPC Gateway Endpoint Behavior: VPC Gateway Endpoints themselves don't introduce specific connection-drain behaviors that would cause this pattern. The endpoint acts as a routing mechanism and shouldn't be the primary cause of your latency variance between high and low TPS periods.

3. CRT vs. Netty Client: Both clients should be capable of maintaining connection pools effectively. The choice between CRT and Netty is typically based on performance characteristics at scale rather than solving cold-connection issues. The underlying problem—connection pool management during low traffic—exists regardless of the HTTP client implementation.

4. Keeping Connections Warm: Consider implementing application-level strategies such as periodic health checks or background requests during low-traffic periods to keep the connection pool active. This "synthetic traffic" approach can prevent connections from going cold without requiring SDK-level modifications.

Additional Recommendations:

• Configure appropriate API timeouts (apiCallAttemptTimeout and apiCallTimeout) to ensure your application fails fast during network issues rather than waiting and accumulating latency.
• Monitor and log request IDs for slow requests, as this information is valuable if you need to escalate with AWS Support.
• Remember that the SuccessfulRequestLatency metric only measures DynamoDB service-internal latency—client-side activity and network trip times aren't included. Enable SDK latency metric logging to get a complete picture of where latency is occurring in your request path.

For streaming operations or any SDK operations, ensure you're properly closing resources to prevent connection pool exhaustion, which could exacerbate latency issues during traffic fluctuations.
Sources
Troubleshooting latency issues in Amazon DynamoDB - Amazon DynamoDB
Best practices for using the AWS SDK for Java 2.x - AWS SDK for Java 2.x