- Newest
- Most votes
- Most comments
For others benefit, just wanted to highlight, this issue was coming when I am creating the connection from AWS glue console and selecting mysql (or postgresql) as the data source. If I choose JDBC as the connection type, I am not facing any such error.
**Till the time, aws glue team looks into the issue, I am accepting this as an answer. Hope this helps the people who are stuck with this issue!
answered 2 years ago
Does your security group on the RDS allow the glue connection?
Because your RDS may be set to public, you will need to use the private IP address of the RDS Instance in the JDBC connection and not the endpoint hostname as GLUE in theory is in a different VPC therefore it will resolve to the Public IP.
The timeout error you're experiencing while creating an AWS Glue Connection for your RDS instance could be due to several factors. Let's address some potential issues and solutions:
-
Network Configuration: Even though your RDS instance is in a public subnet and you've verified network communication with the reachability analyzer, there might still be some network-related issues. Ensure that the security group associated with your RDS instance allows inbound traffic from the Glue service on the appropriate port (typically 3306 for MySQL or 5432 for PostgreSQL).
-
VPC Endpoint: If you're using a VPC, make sure you have a VPC endpoint for AWS Glue. This allows Glue to communicate with your resources within the VPC without going through the public internet, which can help resolve timeout issues.
-
Firewall Rules: Check if there are any firewall rules or network ACLs that might be blocking the connection between Glue and your RDS instance.
-
Connection Properties: Double-check the connection properties, including the JDBC URL, port, and database name. Make sure they are correctly formatted and match your RDS instance details.
-
IAM Permissions: While you've mentioned that your Glue service role has the necessary permissions, it's worth double-checking that it has all required permissions for both RDS and Secrets Manager. The fact that Secrets Manager is involved even when using username-password authentication is expected behavior, as AWS Glue securely stores connection credentials.
-
RDS Instance Status: Verify that your RDS instance is in an available state and not undergoing maintenance or other operations that might affect its accessibility.
-
Increase Timeout: If possible, try increasing the connection timeout setting for the Glue Connection. However, this option might not be directly available in the console and may require using the AWS CLI or SDK.
-
Use EC2 for Testing: Launch an EC2 instance in the same VPC and subnet as your RDS instance, and try connecting to the RDS from there. This can help isolate whether the issue is specific to Glue or a more general networking problem.
If the issue persists after checking these points, you may need to contact AWS Support for further assistance, as there could be underlying service-specific issues or conflicts that are not immediately apparent.
Sources
Troubleshooting connection issues in AWS Glue - AWS Glue
Creating connections for connectors - AWS Glue
AWS Glue job failed while connecting to aws glue cause of timeout | AWS re:Post
I already tried these suggestions and none of them worked. I created an EC2 instance in the same vpc-subnet and I am able to connect to my RDS instance, from there.
Yes my RDS security group does allow glue connection (glue security group). I tried with turning off public accessibility to NO for RDS, as well. No luck, same error.
** Also my understanding is that internal DNS triggers first, which in my opinion should identify the destination first, which in turn means that public accessibility should not matter. please correct me if I am missing out anything here.
Internal DNS resolution is first IF in the same VPC. Glue techincally is not in your VPC but an AWS VPC. Have you tried using the Private IP in the connection?
Looks like, this issue is coming when I am creating the connection from AWS glue console and selecting mysql (or postgresql) as the data source. I tried with JDBC connection and it is working fine (with dns name).