By using AWS re:Post, you agree to the Terms of Use

unable to install psycopg2 in glue JUPYTER notebook

0

DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support Defaulting to user installation because normal site-packages is not writeable Looking in indexes: https://aws:****@amazon-590186200215.d.codeartifact.us-east-1.amazonaws.com/pypi/aws-glue/simple/ User for amazon-590186200215.d.codeartifact.us-east-1.amazonaws.com:

asked 6 months ago171 views
1 Answer
0

If you are using a Sagemaker notebook attached to Glue dev endpoint with PySpark kernel, please use the following method to update the libraries.

  1. Update the dev endpoint using aws glue update-dev-endpoint --cli-input-json file://./input.json where input.json looks like
{
       "EndpointName": "YOUR_DEV_ENDPOINT_NAME",
       "CustomLibraries": {
           "ExtraPythonLibsS3Path": "s3://my-bucket/library1.zip,s3://my-bucket/library2.zip,s3://my-bucket/library3.zip"
       },
       "UpdateEtlLibraries": true
  }
  1. You can then verify the new list of libraries using following Python code:
import os
os.listdir('.')

Please note that as of writing this, Sagemaker Jupyter notebooks running PySpark kernel now come with following Python version

3.6.12 (default, May 18 2021, 22:47:55) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

SUPPORT ENGINEER
answered 6 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions