AWS ECS containers having random difficulties connecting to AWS RDS

Question

Hello, we are running containers in ECS and using host/url to specify the database for a Rails app. It works on 80% of the instances, and in some other instances, the system complains of a socket connection to mysql error:

A Mysql2::Error::ConnectionError occurred in #:

Can't connect to local MySQL server through socket '/run/mysqld/mysqld.sock' (2)

I can't figure out why this is very random, connection capacity in the RDS is not even at 10%. It's always triggered with the ELB check, randomly. Any direction is appreciated, thanks!

Answer

Hi,

Thank you for contacting us.

Based on the error message we could see that there is some local database dependency in the application due to which it it trying to connect to the database in localhost rather than RDS. If there are no other congestion noted from the container side, please check from the application point to understand more on this.

Thank you.

Answer

Hello, dependency health check is not a best practices as it can flag your task as irresponsive even in case of an other problem on the database. cf https://aws.amazon.com/builders-library/implementing-health-checks/

Does you code has a fallback on the local socket in case of connection error on the url of the DB ?
There are several factors that can make connection to fail on the DB: the number of allowed connections, or the network bandwith that are different dependent of the instance size : https://aws.amazon.com/rds/instance-types/

AWS ECS containers having random difficulties connecting to AWS RDS

Relevant content