Intermittent slow connection fargate service connected to application load balancer

Question

I have a multi-tenant setup consisting of a backend and a store application. The backend routes requests to the appropriate tenant database based on the incoming request from the store app. Both applications are containerized within a single task, and the Fargate service is registered with a target group, which is attached to an Application Load Balancer (ALB). Both apps use Node.js as their server runtime—Express.js for the backend and Next.js for the store app.

Recently, the store app has been experiencing intermittent slow connections when fetching data from database with large data. This issue typically begins to appear around 4–5 days after deployment. However, CPU and memory utilization for both the Fargate service and the database appear healthy, and there are no container exit codes.

I'm aware that the default idle timeout for the ALB is 60 seconds, and Node.js’s HTTP server has a keepAliveTimeout of 5 seconds. I tested this by creating a dummy API with a 16-second delay, which returned a successful 200 OK response.

My current challenge is reproducing the issue and identifying which component might be causing the intermittent slow connections. Any insights would be appreciated—thanks!

re:Post Agent · Answer

Based on the information provided, there are several potential causes for the intermittent slow connections you're experiencing. Here are some insights and suggestions to help you troubleshoot the issue:

1. Connection Draining: The intermittent nature of the problem, occurring 4-5 days after deployment, could be related to connection draining. When tasks are replaced or updated, the ALB needs to drain existing connections. You might want to optimize the ECS_CONTAINER_STOP_TIMEOUT parameter. Setting this to a lower value (e.g., 2 seconds) could help speed up the connection draining process.

2. Target Group Health Checks: Ensure that your target group health check settings are optimized. Misconfigured health checks can lead to intermittent connectivity issues.

3. Network Configuration: Verify that your VPC, subnets, security groups, and network ACLs are correctly configured to allow traffic between all components of your setup.

4. Database Connection Pool: Although you mentioned the database appears healthy, consider reviewing your database connection pool settings. If connections are not being properly managed or released, it could lead to slowdowns over time.

5. Application-Level Issues: Since both your backend and store app are in the same task, investigate if there are any memory leaks or resource contention issues within your Node.js applications. Long-running processes or improper handling of large datasets could contribute to degraded performance over time.

6. ALB Metrics: Monitor the TargetResponseTime CloudWatch metric for your ALB. This can help you identify if the slowdown is occurring at the application level or if it's related to the ALB itself.

7. Container-Level Metrics: Use CloudWatch Container Insights or log into the containers directly to gather more detailed metrics about the application's performance.

8. Load Testing: Consider performing load tests that simulate your production environment to try and reproduce the issue. This can help identify bottlenecks or scaling issues that may not be apparent under normal conditions.

9. Logging and Tracing: Implement comprehensive logging and potentially distributed tracing to get a clearer picture of where the slowdowns are occurring in your request flow.

10. ALB Access Logs: Enable and analyze ALB access logs to get more detailed information about the requests being processed.

Remember, intermittent issues can be challenging to diagnose. A systematic approach, starting with the application and moving outward to the infrastructure, can help isolate the problem. Consider implementing more granular monitoring and alerting to catch these issues earlier and gather more data when they occur.  
**Sources**  
[Troubleshoot high response times and ECS latency issues | AWS re:Post](https://repost.aws/knowledge-center/ecs-troubleshoot-high-response-times)  
[Optimize load balancer connection draining parameters for Amazon ECS - Amazon Elastic Container Service](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/load-balancer-connection-draining.html)  
[Troubleshoot your Application Load Balancers - Elastic Load Balancing](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html)

Intermittent slow connection fargate service connected to application load balancer

Add your answer

Relevant content