Sporadic no response from dynamodb request

Question

Hi dynamodb experts ;-)  
  
I have an infrequent issue:  
Sometimes (one of thousands or more of requests) a dynamodb call (no matter which table or what kind of request, get, query, put, del, etc...) is not responding (in time).  
  
I call dynamodb from my lambda functions and track it like this (I track the calls with iopipe):

```
  const ddb = new AWS.DynamoDB.DocumentClient({ region: settings.region });

  const params = {
    TableName: 'my-table-for-user-projects',
    Key: {
      projectId: projectId,
      userId: userId
    }
  };

  settings.track('database').start();
  ddb.get(params, (err, ret) => {
    settings.track('database').end();
    if (err) return callback(err);
    callback(null, ret.Item);
  });
```

in iopipe I see "NO TRACE END DETECTED" for my custom metric and also for the dynamodb call, i.e:  
  
_REQUEST HEADERS_  
_Authorization_  
AWS4-HMAC-SHA256 Credential=ASIA************G3Y/20190222/eu-west-1/dynamodb/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-target, Signature=ca368c**************************************************bb6ac670  
_Content-Length_  
160  
_Content-Type_  
application/x-amz-json-1.0  
_Host_  
dynamodb.eu-west-1.amazonaws.com  
_User-Agent_  
aws-sdk-nodejs/2.406.0 linux/v8.10.0 exec-env/AWS_Lambda_nodejs8.10 callback  
_X-Amz-Content-Sha256_  
132956**************************************************220b3d9f  
_X-Amz-Date_  
20190222T145758Z  
_X-Amz-Target_  
DynamoDB_20120810.GetItem  
_x-amz-security-token_  
FQoGZXIvY*****************************************************xsfm9zKPPyv+MF  
_No Response Headers_  
  
-----  
  
_Has there been any internal changes to dynamodb? Or any known network issues? Not resolving the correct dns name or similar?_  
  
**Maybe other useful hints:** Have these timeouts since 1-2 weeks and I switched the dynamodb plan to _on-demand_ since last month. And also important: seems there are no big workload peaks when the timeouts are triggered (no special dynamodb capacity increase and no throttled request).  
  
I hope anyone can help, because my customers starts to notice it. :-(  
  
Have a nice day.

Answer

Thanks for your reply.  
I see the same stuff you mentioned for thundra also in iopipe...  
See screenshot here: https://twitter.com/adrirai/status/1099224045365669888  
My setup with lambda + dynamodb is like for more than 2 years now... and the timeouts just appeared a couple of weeks ago... (but as said, really just very rare. Perhaps some dynamodb nodes react differently?)

Answer

Changing the default http timeout in the aws-sdk seems to solve my problem.  
  
AWS.config.update({httpOptions: {timeout: 5000}}); // default is 120000 => 2 mins  
// or  
const ddb = new AWS.DynamoDB.DocumentClient({ region: settings.region, httpOptions: { timeout: 5000 } }); // default is 120000 => 2 mins

Answer

Hi adrai,  
  
I am not sure if this is the case for your but we have encountered a similar problem recently. Basically, we were experiencing an unexpected delay (100ms to 300ms, depending on the memory allocated for the corresponding Lambda) for some dynamodb calls we made in our Python function and this was happening randomly.   
  
In our case since we had a bigger timeout value set in our Lambda, functions didn't timeout. However, we were using Thundra to monitor our Lambda function and it has an AWS SDK integration shows the individual dynamodb call's duration. We saw in the Thundra console that some random dynamodb calls experience an extra latency.  
  
We eventually found out the reason was the connection timeout value that AWS SDK uses while making API calls. It was set to the 60s by default for Python AWS SDK. Basically, whenever there is a gap between two dynamodb calls greater than the 60s, AWS SDK creates a new connection and that was causing unexpected latency I have mentioned above. As long as the time between two dynamodb calls does not exceed the 60s, AWS SDK uses the same connection, so there is no extra latency for creating a new connection. Also, note that this is not something related to cold-start. Even if your Lambda is warm, once the connection that AWS SDK uses to make API calls is timed out; same latency is occurring again.  
  
I don't know if this was the exact issue you faced with, but maybe that gives you a clue about one of the possible causes. Also, I suggest you have a look at Thundra (https://www.thundra.io). It helps a lot to deal with that kind of problems.

Sporadic no response from dynamodb request

Relevant content