unable to install psycopg2 in glue JUPYTER notebook

Question

DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://aws:****@amazon-590186200215.d.codeartifact.us-east-1.amazonaws.com/pypi/aws-glue/simple/
User for amazon-590186200215.d.codeartifact.us-east-1.amazonaws.com:

Answer

If you are using a Sagemaker notebook attached to Glue dev endpoint with PySpark kernel, please use the following method to update the libraries.

1. Update the dev endpoint using `aws glue update-dev-endpoint --cli-input-json file://./input.json` where input.json looks like 
```
{
       "EndpointName": "YOUR_DEV_ENDPOINT_NAME",
       "CustomLibraries": {
           "ExtraPythonLibsS3Path": "s3://my-bucket/library1.zip,s3://my-bucket/library2.zip,s3://my-bucket/library3.zip"
       },
       "UpdateEtlLibraries": true
  }
```

2.  You can then verify the new list of libraries using following Python code:
```
import os
os.listdir('.')
```
Please note that as of writing this, Sagemaker Jupyter notebooks running PySpark kernel now come with following Python version

`3.6.12 (default, May 18 2021, 22:47:55) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]`

unable to install psycopg2 in glue JUPYTER notebook

Relevant content