DAX client for JS: EMFILE issues

Question

Hello,

We're using the[ amazon-dax-client](https://www.npmjs.com/package/amazon-dax-client) (latest version: 1.2.9) to query DAX (10-20k calls per execution) inside a lambda.

We've initialized it as per documentation, outside of handler functions.

```
  const dax = new AmazonDaxClient({ endpoints: [DAX_ENDPOINT], region: process.env.AWS_REGION });
  return new AWS.DynamoDB.DocumentClient({ service: dax });

```

We're getting the following issue regularly when we make calls.  We suspect the dax client is creating many sockets when we parallelize our calls to DAX. 
```
Error: ConnectionException: connect EMFILE 100.64.4.159:8111 - Local (undefined:undefined)
```

Any ideas on fixing this? We've tried upgrading lambda memory / monitoring DAX CPU (seems fine) and we're looking into limiting the http agent.maxSockets/maxTotalSockets to a lower number to force reuse.

Cheers,
Paul

Answer

Hello,

Thank you for using AWS DAX service and reaching out to us. I understand that you are facing EMFILE error.

Generally the EMFILE error code indicates that the execution environment (aka instance) running your Lambda function code isn't allowing any more file handles/sockets. This can happen when the maximum number of file descriptors allowable on the system has been reached, and requests for another descriptor cannot be fulfilled until at least one has been closed or released. Lambda imposes a hard quota of 1024 for file descriptors [1]. Another possible cause for this error could be that you are exceeding the amount of ephemeral storage memory configured for your Lambda function.

One way to get system level metrics such as the number of open file descriptors and whether the ephemeral storage is being used up is to use Lambda Insights. Lambda Insights collects, aggregates, and summarizes diagnostic information and metrics. Use this link [2] for more information about enabling and using Lambda Insights.

Additionally, I would also recommend you check this 3rd party article [3] which discusses how to debug this error by using CloudWatch Log Insights and enabling Lambda Insights. While the 3rd party article discusses fixing the error using the "AWS_NODEJS_CONNECTION_REUSE_ENABLED" environment variable, Lambda best practices recommends using a keep-alive directive in the fucnction code to maintain persistent connections. Please see Lambda best practices here [4] and another 3rd party article [5] demonstrating the results of using a keep-alive directive in node.js Lambda functions.

For more detailed assistance, kindly reach out to  AWS Support [6], along with your issue/use case in detail and share relevant AWS resource names and logs, and we will be happy to troubleshoot accordingly.

References:

[1] Lambda Quotas - https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html#function-configuration-deployment-and-execution
[2] Using Lambda Insights - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights.html
[3] https://bahr.dev/2021/06/03/lambda-emfile/
[4] Best practices for working with AWS Lambda functions - https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html#function-code
[5] https://hackernoon.com/lambda-optimization-tip-enable-http-keep-alive-6dc503f6f114
[6] https://docs.aws.amazon.com/awssupport/latest/user/case-management.html#creating-a-support-casehttps://docs.aws.amazon.com/awssupport/latest/user/case-management.html#creating-a-support-case

DAX client for JS: EMFILE issues

Relevant content