Connect to RDS postgres database in GLUE script

Question

I would like to connect to the RDS postgres database with Glue script written in scala, I have a postgres driver located in S3 and added in script in Dependent JARs path. After running the script it fails with the message `Glue ETL Marketplace - Retrieved no ETL connector jars, this may be due to no marketplace/custom connection attached to the job or failure of downloading them, please scroll back to the previous logs to find out the root cause. ` I cannot find any related logs helping me to understand why there is a problem and I receive a Compilation error.

Answer

Hi,

any specific reason why you are not using the [native Glue connection](https://docs.aws.amazon.com/glue/latest/dg/connection-using.html) which supports RDS PostgreSQL?

Glue 2.0 uses the driver 42.1.0 and Glue 3.0 uses 42.2.18 (see [in this section of the documentation](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html#aws-glue-programming-etl-connect-jdbc) or [in this other one](https://docs.aws.amazon.com/glue/latest/dg/populate-add-connection.html)).

If you really need to bring your own connector, have you followed [the documentation](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html#aws-glue-programming-etl-connect-jdbc) or [this blog post](https://aws.amazon.com/blogs/big-data/building-aws-glue-spark-etl-jobs-by-bringing-your-own-jdbc-drivers-for-amazon-rds/) as an example?

can you share an anonymised version of the code snippet that contains the setup for the  `customJDBCDriverJsonOptions` ?

hope this helps

Connect to RDS postgres database in GLUE script

相關內容