Elastic Beanstalk loses connection to DynamoDB intermittently

Question

I have an Elastic Beanstalk app that is a rest API running on NodeJS. The app runs in us-west-1 as do all the DynamoDB tables. The only thing the app does in a different region is SMS but that region is set specifically for the scope of sending the message and it is verified not to overwrite the normal config.

In the last couple months we have had these cases occurring where API stops responding and the error in the logs is:

```
ResourceNotFoundException: Requested resource not found
    at Request.extractError (/var/app/current/node_modules/aws-sdk/lib/protocol/json.js:80:27)
    at Request.callListeners (/var/app/current/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/var/app/current/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/var/app/current/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/var/app/current/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/var/app/current/node_modules/aws-sdk/lib/state_machine.js:14:12)
   at /var/app/current/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request. (/var/app/current/node_modules/aws-sdk/lib/request.js:38:9)
   at Request. (/var/app/current/node_modules/aws-sdk/lib/request.js:688:12)
   at Request.callListeners (/var/app/current/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'ResourceNotFoundException',
  '[__type]': 'See error.__type for details.',
  time: 2025-04-01T19:03:01.100Z,
  requestId: '',
  statusCode: 400,
  retryable: false,
  retryDelay: 322.113315250136
}
```
Searching on this error, it is often caused by a mismatch in region or credentials or the table is not ready. In this case I can eliminate credentials (only one account) and I know this table exists in the region I'm calling. One important detail: if I restart the app servers on my EB app, everything is fine again for a week or two. I would assume if I had region or credential problems it would persist even after restarting.

The only other thing I can think of is I am logging too much. I had a lot of verbose logging initially that I am going to remove, but would too much logs actually cause this particular problem?

re:Post Agent · Accepted Answer

Based on the information provided, it seems unlikely that the issue is related to region mismatch, credentials, or the table not existing, especially since restarting the app servers temporarily resolves the problem. The intermittent nature of the issue and its resolution through restarts suggest a different underlying cause.

One possibility to consider is that your application might be experiencing connection issues with DynamoDB. This could be due to network instability, connection pool exhaustion, or improper handling of DynamoDB client connections.

Here are a few suggestions to investigate and potentially resolve the issue:

1. Connection pooling: Ensure you're using proper connection pooling with the DynamoDB client. Creating a new client for each request can lead to resource exhaustion.

2. Error handling and retries: Implement robust error handling and retry mechanisms in your application. The AWS SDK for Node.js has built-in retry capabilities that you should leverage.

3. Timeouts: Check if you have appropriate timeout settings for your DynamoDB operations. Long-running queries might be timing out.

4. Monitoring: Set up detailed monitoring for your Elastic Beanstalk environment and DynamoDB tables. This can help identify resource constraints or performance issues.

5. SDK version: Ensure you're using the latest version of the AWS SDK for Node.js. Older versions might have bugs that could cause intermittent connection issues.

6. VPC configuration: If your Elastic Beanstalk environment is in a VPC, verify that the network configuration allows proper communication with DynamoDB.

Regarding your concern about logging, while excessive logging can impact performance, it's unlikely to directly cause the specific "ResourceNotFoundException" error you're seeing. However, reducing verbose logging is generally a good practice for improving overall application performance.

To further diagnose the issue, you might want to implement more targeted logging around your DynamoDB operations, focusing on connection establishment and error cases. This could help identify patterns or specific conditions that lead to the connection losses.

If the problem persists after implementing these suggestions, you may want to consider reaching out to AWS support for more in-depth investigation, as there could be underlying infrastructure or service-specific issues at play.  
**Sources**  
[Migrating to DAX Node.js SDK V3 - Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DAX.client.run-application-nodejs-3-migrating.html)  
[Deploying a Node.js application with DynamoDB to Elastic Beanstalk - AWS Elastic Beanstalk](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/nodejs-dynamodb-tutorial.html)

Elastic Beanstalk loses connection to DynamoDB intermittently

Add your answer

Relevant content