504 timeout error on EC2 instance inside ECS cluster on same time daily

Question

We are facing very weird situation in ECS cluster.  ECS tasks minimum  is 6 and max count is 40.  
Task defination : 
`"cpu": "1600",
"memory": "2048",`

Every day at same time people are login to system or application one or two instance getting higher utilisation 90% so we are getting 504 timeout error for each application API request. We check container log didn't find any error causing issue. Request which is getting timeout later if we tried it works sometime. So nothing is do with particular request causing issue.

What could be issue ? We are struggling to get rid of this issue from 5 months.

Answer

If you have ALB logging enabled then I would use the logs to find which container is returning the 504

Then dig deeper into the application. It may be the container is overloaded and out of available http connections at that time or not enough threads.

You’d need to look at tuning your app.

504 timeout error on EC2 instance inside ECS cluster on same time daily

Relevant content