EMR serverless spark jobs connection with postgresql

0

Hi, I want to run a job on EMR serverless which reads and writes data from postgresql. I downloaded the jar file and pushed it to s3 and set "spark.jars" in Spark properties in management console. However the job is still failing.

Thank you, Muthu

  • More details are needed including what the error message is when the job fails. That said, if you are connecting to postgres, make sure your Serverless application is created in a VPC and that the security groups have access to the database. Reachability analyzer can be used to debug network connectivity issues.

  • Hi Muthu,

    Can you please share what error you are getting and snippet of your code if possible to see how you are trying to connect?

muthu
gefragt vor einem Jahr1058 Aufrufe
1 Antwort
2
Akzeptierte Antwort

Thank you all for taking your time to reply, I solved the issue by following Dacort's comment and setting up my Serverless application inside a VPC whose security groups have access to the database. This is my code snippet

sample_data = spark.read.format("jdbc").options(

url='jdbc:postgresql://<sample-name>.<region-name>.rds.amazonaws.com/dev',

dbtable='public."<sample-name>"',

user='<sample-user>',

password='<sample-pass>',

driver='org.postgresql.Driver').load()

I was getting this error :

Caused by: java.net.SocketTimeoutException: connect timed out : org.postgresql.util.PSQLException: The connection attempt failed.
muthu
beantwortet vor einem Jahr

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen