importing psycopg2 to glue job fails

0

Im using aws Glue in order to transfer from pg to s3. I wrote a script which seems to work.

Im trying to import psycopg2 to my script, And I keep getting the following error: "ModuleNotFoundError: No module named 'psycopg2'".

I tried adding the module both as a zip and whl to the s3 and then point the as a 'Python library path'.

It is important to note that my zip file is including the following folders:

  • psycopg2

  • psycopg2_binary.libs

  • psycopg2_binary-2.9.7.dist-info

I also tried adding it as a Job Parameter with all this options:

--additional-python-modules | psycopg2

--additional-python-modules | psycopg2==2.9.7

--additional-python-modules | psycopg2-binary

--additional-python-modules | psycopg2-binary==2.9.7

--extra-py-files | psycopg2

--extra-py-files | psycopg2==2.9.7

--extra-py-files | psycopg2-binary

--extra-py-files | psycopg2-binary==2.9.7

still nothing seems to work.

itamar
asked 8 months ago351 views
1 Answer
0

When you say "using Glue", I guess you mean "Glue Shell" otherwise it wouldn't make sense to use a Python library for that. --additional-python-modules psycopg2-binary does work, check your logs, maybe you are running in a VPC and don't have Pypi access or you are using the old shell version.

profile pictureAWS
EXPERT
answered 8 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions