unable to install psycopg2 in glue JUPYTER notebook

If you are using a Sagemaker notebook attached to Glue dev endpoint with PySpark kernel, please use the following method to update the libraries.

1. Update the dev endpoint using aws glue update-dev-endpoint --cli-input-json file://./input.json where input.json looks like

{
       "EndpointName": "YOUR_DEV_ENDPOINT_NAME",
       "CustomLibraries": {
           "ExtraPythonLibsS3Path": "s3://my-bucket/library1.zip,s3://my-bucket/library2.zip,s3://my-bucket/library3.zip"
       },
       "UpdateEtlLibraries": true
  }

2. You can then verify the new list of libraries using following Python code:

import os
os.listdir('.')

Please note that as of writing this, Sagemaker Jupyter notebooks running PySpark kernel now come with following Python version

3.6.12 (default, May 18 2021, 22:47:55) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]