Experiencing latency on DynamoDB due to CredentialsRequestTime

0

Hi everyone, I have an ECS cluster running and one of service container hits DynamoDB with GETITEM request. Most of these requests get completed within 10 ms but every 30 minutes or so I can see latency going over 2+ seconds. I am using Java SDK to hit the DynamoDB. Intially I thought that this might happen due to connections creation latency so I am using keep alive connection now but still facing same issue. To get more depth I started listening to request metrics and pushed them on to Kibana. Below is a sample request metric :

{
  "HttpRequestTime": 158.081,
  "RequestSigningTime": 15.361,
  "CredentialsRequestTime": 2184.899,
  "ResponseProcessingTime": 17.205,
  "totalTime": 2707.05,
  "responseCode": 200,
  "ClientExecuteTime": 2707.003,
  "consistentRead": false,
  "RequestMarshallTime": 198.897,
  "ApiCallLatency": 242.628,
  "HttpClientReceiveResponseTime": 22.391,
  "HttpClientSendRequestTime": 2.166
}

As you can see that totalTime taken by GETITEM request is 2707 and CredentialsRequestTime is taking most of the time. So how can I minimize this?

asked 2 years ago630 views
1 Answer
1

Hi,

Ideally if you application is latency sensitive then you should not allow it to wait for that length of time. Given that most of your requests are sub 10ms, it doesn't make sense for your client wait for almost 3000ms.

Firstly you should enable SDK Metrics or AWS X-Ray to understand if the increase in latency is caused by DynamoDB. If so, this is where tuning the client comes into play, you should fail-fast and retry: This Blog is very useful to grasp the concept of client tuning. Your configuration will ultimately depend on your application needs, but reducing the time you wait on a response will greatly reduce the outlier issue you are experiencing.

profile pictureAWS
EXPERT
answered 2 years ago
  • So I finally managed to fix this by extending AWSCredentialsProvider class and creating a custom credentials provider. I simply keep credentials cached and refresh them every 5 minutes.

  • @GhostFoxGod Could you please share your code snippet/repo for maintaining the credentials with required dependency.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions