Is it possible to run a Glue Crawler on an Aurora Postgres 15 Serverless v2 database?
Following this issue here:
https://github.com/awslabs/aws-advanced-jdbc-wrapper/issues/653
For the last few days I have been trying to run a Glue Crawler on an Aurora 15.3 Serverless v2 database.
I set up my Connections as follows:
- Connection URL: jdbc:postgresql://<endpoint>.<region>.rds.amazonaws.com:5432/<database>
- Driver Path: s3://aws-glue-<bucketName>/postgresql-42.6.0.jar
- Driver Class Name: org.postgresql.Driver
- I got the driver from here : https://jdbc.postgresql.org/download/
- Credentials: Username and Password
- Require SSL Connection: False
- Security Groups: Allowing all inbound and outbound traffic
- Subnet: My crawler subnet is in one of the subnet groups for my RDS instance.
What I've tried:
- Using an Admin Role:
The glue crawler is using an admin role that has permissions over all of my AWS environment. I tried using an [IAM PassRole ]
(https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_passrole.html) as well just to be sure.
- Parameter Store
Some folks have mentioned that adding a parameter group and then adding changing the password encryption from SHA-256 to md5
Sample here
It seems the parameter store has an option for RDS Postgres 15 but not for RDS Aurora Postgres 15
- Adding managed prefix to inbound rule for security group
Crawler still errors out with "Internal Service Issue" and
Caused by: com.amazonaws.services.glue.exceptions.InvalidInputException: Testing connections with custom drivers is not currently supported.
at com.amazonaws.glue.jobexecutor.commands.jdbc.JdbcCommandArgumentsUtil.handleCustomDrivers(JdbcCommandArgumentsUtil.java:338)
at com.amazonaws.glue.jobexecutor.commands.jdbc.JdbcCommandArgumentsUtil.getJdbcArgumentsFromCatalog(JdbcCommandArgumentsUtil.java:201)
at com.amazonaws.glue.jobexecutor.commands.jdbc.JdbcCommandArgumentsUtil.getJdbcArgumentsRetry(JdbcCommandArgumentsUtil.java:118)
Unlike the tester, the crawler should accept custom drivers, check with support what is the error that prevents the crawler from running or test the connection using code