Seeking Solutions for Intermittent High Latency Issues with AWS AppSync in Node.js Server and Lambda Function

We are encountering intermittent high latency issues with AWS AppSync in two distinct setups: a Node.js API server deployed on an EC2 instance and a Node.js application running within an AWS Lambda function. Both environments utilize the aws-appsync library to query data from DynamoDB. It's important to note that our AppSync setup does not use a private API. The EC2 setup includes SSL authentication and load balancing managed by Nginx, configured with the following specifications:

Node.js Version: 13.6.0 EC2 Instance Type: m5a.2xlarge Operating System: Amazon Linux 2

EC2 Instance Setup: Processes approximately 145,000 requests per hour, which includes around 100,000 GraphQL queries to AWS AppSync. Experiences intermittent latency, with 10 to 150 queries taking over 30 seconds to connect to AppSync daily at various times, with some days showing no errors. The average request rate is ~40 requests/second, peaking at 80 requests/second. Metrics for AppSync and DynamoDB indicate the highest query latency at 30ms, with no errors logged.

Specific Incident:

On March 30, 2024, at 22:35:58 +0900, a query was sent from the EC2 server but only reached aws AppSync at 22:39:32.999 +0900. It was processed within approximately 5ms, and the server received a response at 22:39:33 +0900, resulting in a total delay of 214 seconds. All other queries were successful with latency below 500ms.

Lambda Function Setup:

Handles around 800 requests per hour, which includes around 900 GraphQL queries to AWS AppSync. Experienced a similar latency issue to the EC2 instance at the same timestamp. Despite detailed metrics analysis for both AppSync and DynamoDB, indicating no errors and a maximum query delay of 30ms, we face these unexplained latency spikes that occur sporadically, with some days not recording any errors, making diagnosis challenging.

We are reaching out to the AWS community for advice, insights, or potential solutions to address these intermittent high latency issues. Could network issues within EC2 or between EC2/Lambda and AppSync be the cause? Or might there be overlooked aspects in our configuration or code?