Is it possible to run a Glue Crawler on an Aurora RDS Postgres 15 Serverless v2 database?

0

Is it possible to run a Glue Crawler on an Aurora Postgres 15 Serverless v2 database?

Following this issue here: https://github.com/awslabs/aws-advanced-jdbc-wrapper/issues/653

For the last few days I have been trying to run a Glue Crawler on an Aurora 15.3 Serverless v2 database.

I set up my Connections as follows:

  • Connection URL: jdbc:postgresql://<endpoint>.<region>.rds.amazonaws.com:5432/<database>
  • Driver Path: s3://aws-glue-<bucketName>/postgresql-42.6.0.jar
  • Driver Class Name: org.postgresql.Driver
  • I got the driver from here : https://jdbc.postgresql.org/download/
  • Credentials: Username and Password
  • Require SSL Connection: False
  • Security Groups: Allowing all inbound and outbound traffic
  • Subnet: My crawler subnet is in one of the subnet groups for my RDS instance.

What I've tried:

  1. Using an Admin Role:

The glue crawler is using an admin role that has permissions over all of my AWS environment. I tried using an [IAM PassRole ]

(https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_passrole.html) as well just to be sure.

  1. Parameter Store Some folks have mentioned that adding a parameter group and then adding changing the password encryption from SHA-256 to md5

Sample here

It seems the parameter store has an option for RDS Postgres 15 but not for RDS Aurora Postgres 15

  1. Adding managed prefix to inbound rule for security group

Crawler still errors out with "Internal Service Issue" and

Caused by: com.amazonaws.services.glue.exceptions.InvalidInputException: Testing connections with custom drivers is not currently supported. at com.amazonaws.glue.jobexecutor.commands.jdbc.JdbcCommandArgumentsUtil.handleCustomDrivers(JdbcCommandArgumentsUtil.java:338) at com.amazonaws.glue.jobexecutor.commands.jdbc.JdbcCommandArgumentsUtil.getJdbcArgumentsFromCatalog(JdbcCommandArgumentsUtil.java:201) at com.amazonaws.glue.jobexecutor.commands.jdbc.JdbcCommandArgumentsUtil.getJdbcArgumentsRetry(JdbcCommandArgumentsUtil.java:118)

  • Unlike the tester, the crawler should accept custom drivers, check with support what is the error that prevents the crawler from running or test the connection using code

No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions