ETL Job connection timeout exception when running,

Question

I created connections to RDS and Redshift in AWS Glue connections. Both connections worked when running Test Connection and Crawler. I see that once in a while RDS connection test fails throwing a Network Failure message.

When I run the ETL Jobs using the connections it throws "org.postgresql.util.PSQLException: The connection attempt failed. (Caused by java.net.SocketTimeoutException: connect timed out)" . Not sure why its happening.

- The role used for connection has all the access needed for Glue, Rds, and Redshift.
- The security group permission is added and configured correctly so the Glue can access data.

I use a RDS aurora-postgresql-13 rds instance.

Accepted Answer

Changed my RDS instance type to a non-ARM processor and created a new connection with the same configuration and it worked.

Answer

When creating a connection in glue that requires access to an RDS on a VPC you will need to set a subnet that’s routable to the RDS instance.

For this you need to define the subnet and security group for the glue job. Have you enabled outbound rules on this security group?

The RDS instance also has a security group which will need to allow inbound access from the glue job.

Both subnets the glue job and Rds instance is hosted on requires routes to be able to route traffic to one another.

Also ensure no NACLs block access either.

ETL Job connection timeout exception when running,

Contenido relevante