Experiencing latency on DynamoDB due to CredentialsRequestTime

Question

Hi everyone,
I have an ECS cluster running and one of service container hits DynamoDB with GETITEM request. Most of these requests get completed within 10 ms but every 30 minutes or so I can see latency going over 2+ seconds. I am using Java SDK to hit the DynamoDB. Intially I thought that this might happen due to connections creation latency so I am using keep alive connection now but still facing same issue. To get more depth I started listening to request metrics and pushed them on to Kibana. Below is a sample request metric :

```
{
  "HttpRequestTime": 158.081,
  "RequestSigningTime": 15.361,
  "CredentialsRequestTime": 2184.899,
  "ResponseProcessingTime": 17.205,
  "totalTime": 2707.05,
  "responseCode": 200,
  "ClientExecuteTime": 2707.003,
  "consistentRead": false,
  "RequestMarshallTime": 198.897,
  "ApiCallLatency": 242.628,
  "HttpClientReceiveResponseTime": 22.391,
  "HttpClientSendRequestTime": 2.166
}
```
As you can see that totalTime taken by GETITEM request is 2707 and CredentialsRequestTime is taking most of the time. So how can I minimize this?

Answer

Hi,

Ideally if you application is latency sensitive then you should not allow it to wait for that length of time. Given that most of your requests are sub 10ms, it doesn't make sense for your client wait for almost 3000ms.

Firstly you should enable SDK Metrics or AWS X-Ray to understand if the increase in latency is caused by DynamoDB. If so, this is where tuning the client comes into play, you should fail-fast and retry: [This Blog](https://aws.amazon.com/blogs/database/tuning-aws-java-sdk-http-request-settings-for-latency-aware-amazon-dynamodb-applications/) is very useful to grasp the concept of client tuning. Your configuration will ultimately depend on your application needs, but reducing the time you wait on a response will greatly reduce the outlier issue you are experiencing.

Experiencing latency on DynamoDB due to CredentialsRequestTime

Relevant content