Cannot seem to import psycopg2 in AWS Glue 3.0
I have tried a myriad of ways to get psycopg2 imported into a script.
- import psycopg2 (on it's own as psycopg2 is listed as available in Glue 3.0 when I print(help("modules"))
- Downloaded both 64-bit and 32-bit and tried both in Python library path (individually) - log actually says these loaded but import still fails
- Added --additional-python-modules as a parameter and tried both with and without a specific version - I think this fails on my because of network security implemented (no downloading from external sources).
I will get the error "ModuleNotFoundError: No module named 'psycopg2._psycopg'" or "ModuleNotFoundError: No module named 'psycopg2'". The former is when I use the whl objects. The latter is when I try to use the module that appears to be available by default.
Please try using the below under job parameters of your Glue 3.0 job.
psycopg2 installation appears to be failing due to a missing dependency as shown below and the developer is suggesting to use psycopg2-binary instead
pg_config executable not found. pg_config is required to build psycopg2 from source. If you prefer to avoid building psycopg2 from source, please install the PyPI 'psycopg2-binary' package instead.
How do you update packages on AWS Glue?asked 8 days ago
AWS Glue - Read a 'local' file in Pythonasked 12 days ago
unable to install psycopg2 in glue JUPYTER notebookasked 3 months ago
How can i run SQL statements in my AWS Glue Scriptasked 2 months ago
Call a stored procedure in Redshift from GlueAccepted Answerasked 3 years ago
Psycopg2 on AWS Lambda not connecting to RDS databaseasked 6 months ago
Using External Python Packages on AWS Glue.asked 6 months ago
Connecting to Redshift with Python - Operation Timed Outasked 4 months ago
Cannot seem to import psycopg2 in AWS Glue 3.0asked 2 months ago
No module named 'psycopg2'asked 2 years ago