unable to install psycopg2 in glue JUPYTER notebook


DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support Defaulting to user installation because normal site-packages is not writeable Looking in indexes: https://aws:****@amazon-590186200215.d.codeartifact.us-east-1.amazonaws.com/pypi/aws-glue/simple/ User for amazon-590186200215.d.codeartifact.us-east-1.amazonaws.com:

asked 2 years ago832 views
1 Answer

If you are using a Sagemaker notebook attached to Glue dev endpoint with PySpark kernel, please use the following method to update the libraries.

  1. Update the dev endpoint using aws glue update-dev-endpoint --cli-input-json file://./input.json where input.json looks like
       "EndpointName": "YOUR_DEV_ENDPOINT_NAME",
       "CustomLibraries": {
           "ExtraPythonLibsS3Path": "s3://my-bucket/library1.zip,s3://my-bucket/library2.zip,s3://my-bucket/library3.zip"
       "UpdateEtlLibraries": true
  1. You can then verify the new list of libraries using following Python code:
import os

Please note that as of writing this, Sagemaker Jupyter notebooks running PySpark kernel now come with following Python version

3.6.12 (default, May 18 2021, 22:47:55) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions