Unable to import/install external library 'psycopg2' in AWS Glue

0

I have a utility of 'psycopg2' library in AWS Glue script. Since it is not included in standard library set, I would need to add externally. I tried to add library name using "--additional-python-modules" in the 'Job Parameters' sub section under 'Job Details' section, but still it said no library found.

Enter image description here

What's the correct way of importing external library? Let's say I have uploaded 'whl' file in my S3 storage and have given the path in the value of 'Job Parameters'. Would I need to specify "!pip install psycopg2" in my Glue script and then "import psycopg2"?

Or what if I include a zip file in the 'Python library path' subsection, can I directly write "import psycopg2". Need assistance with using this library in Glue 4.0.

asked 8 months ago360 views
1 Answer
0

That library has native code dependencies and as indicated in the docs, Glue cannot compile the gcc libraries on installation.
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html#addl-python-modules-support
You could try to find/create a precompiled wheel for that version of Python and Amazon Linux2.
Normally you don't connect from Python to the databases but I guess you want to run some custom command, you can do it using the JDBC driver already included using P4J calls.

profile pictureAWS
EXPERT
answered 8 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions