AWS Glue and RDS Postgres Issue

Question

Hi,
 
I have an RDS postgres db that i want to crawl using aws glue, i already set up the glue crawler job and the glue connection.

But i'm currently getting this error on executing the crawler job.

Test connection failed for connection '***'

Deeksha · Answer

Network Configuration:

Ensure Glue job is in the same VPC and subnet as the RDS instance.
Check security groups to allow inbound/outbound traffic on the PostgreSQL port (default 5432).
Ensure NAT gateway is configured for private subnets if necessary.

Glue Connection Configuration:

Verify the correctness of the JDBC URL in the Glue connection.
Ensure the provided database credentials are accurate.

Database Security:

Ensure the IAM role for the Glue job has permissions to access the RDS instance.
Check that the Postgres database settings allow connections from Glue.

Check Glue Logs:

Review CloudWatch logs for the Glue job to identify specific error details.

Test the Connection Manually:

Use an EC2 instance in the same VPC/subnet to test connectivity to the RDS instance using psql or another client.

SSL Settings:

Verify that SSL is configured correctly if the RDS instance requires SSL for connections.

Glue Crawler Configuration:

Ensure the IAM role used by the Glue Crawler has the necessary permissions.
Consider enabling retries in the Glue job to handle transient network issues.

AWS Glue and RDS Postgres Issue

Relevant content