AWS Glue 3.0 Docker Image - cannot install Jupyter NBExtensions
I have set up my local glue development environment following instructions at https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html and its working . I am trying to extend Jupyter by installing NB Extensions as in https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/install.html
pip command executes fine
pip install jupyter_contrib_nbextensions --user
I encounter an error on this step :
jupyter contrib nbextension install --user
PermissionError: [Errno 13] Permission denied: '/home/glue_user/.jupyter/migrated'
I get a similar error on the Docker Log as well
[W 2022-06-01 07:43:44.083 LabApp] [Errno 13] Permission denied: '/home/glue_user/.jupyter/lab/workspaces'
Any help ?
**Docker Image : **amazon/aws-glue-libs:glue_libs_3.0.0_image_01
**Docker Started using : **
docker run -it -v ~/.aws:/home/glue_user/.aws -v $JUPYTER_WORKSPACE_LOCATION:/home/glue_user/workspace/jupyter_workspace/ -e AWS_PROFILE=$PROFILE_NAME -e DISABLE_SSL=true --rm -p 4040:4040 -p 18080:18080 -p 8998:8998 -p 8888:8888 --name glue_jupyter_lab amazon/aws-glue-libs:glue_libs_3.0.0_image_01 /home/glue_user/jupyter/jupyter_start.sh
Permissions appear to be incorrect.
1st terminal
docker run -it -v ~/.aws:/home/glue_user/.aws -v $JUPYTER_WORKSPACE_LOCATION:/home/glue_user/workspace/jupyter_workspace/ -e AWS_PROFILE=$PROFILE_NAME -e DISABLE_SSL=true --rm -p 4040:4040 -p 18080:18080 -p 8998:8998 -p 8888:8888 --name glue_jupyter_lab amazon/aws-glue-libs:glue_libs_3.0.0_image_01 /home/glue_user/jupyter/jupyter_start.sh
2nd terminal
$ docker exec glue_jupyter_lab ls -la /home/glue_user
total 84
drwx------ 1 glue_user root 4096 Jun 2 14:14 .
drwxr-xr-x 1 root root 4096 Feb 25 05:27 ..
drwxrwxrwx 1 1000 1000 4096 Apr 13 09:44 .aws
drwxr-xr-x 1 glue_user root 4096 Feb 25 05:28 aws-glue-libs
-rw-r--r-- 1 glue_user root 18 Jul 15 2020 .bash_logout
-rw-r--r-- 1 glue_user root 193 Jul 15 2020 .bash_profile
-rw-r--r-- 1 glue_user root 5441 Mar 31 11:22 .bashrc
drwxr-xr-x 1 glue_user root 4096 Jun 2 14:14 .cache
drwxr-xr-x 2 glue_user root 4096 Apr 1 06:10 .certs
drwxr-xr-x 2 glue_user root 4096 Jun 2 14:14 .ipython
drwxr-xr-x 1 glue_user root 4096 Apr 1 06:10 jupyter
drwxr-xr-x 3 root root 4096 Apr 1 06:10 .jupyter
drwxr-xr-x 1 glue_user root 4096 Jun 2 14:14 livy
drwx------ 1 glue_user root 4096 Apr 1 06:42 .local
drwxr-xr-x 1 glue_user root 4096 Jun 2 14:13 spark
drwxr-xr-x 2 glue_user root 4096 Apr 1 06:10 .sparkmagic
drwxr-xr-x 3 glue_user root 4096 Apr 1 06:39 workspace
$
The owner of .jupyter is root.
How to fix
2nd terminal
docker exec -u 0:0 glue_jupyter_lab chown -R glue_user.root /home/glue_user/.jupyter
Now the .jupyter directory will be owned by glue_user.
Yes , but then the chown permission is denied on the docker console. Its asking for the root password , which I do not know
Relevant questions
How to create dynamic dataframe from AWS Glue catalog in local environment?
asked a month agoTesting Crawlers with Glue 3.0 Docker image
asked 3 months agoaws-glue-libs:glue_libs_3.0.0_image_01 image issue
Accepted Answerasked 2 months agoWhat is the most cost efficient and fastest way to start GLUE ETL development
Accepted Answerasked a month agounable to install psycopg2 in glue JUPYTER notebook
asked 3 months agoAWS Glue Image issue
asked 2 months agoAWS GLUE Image certificate related issue
asked 2 months agoAWS Glue 3.0 Docker Image - cannot install Jupyter NBExtensions
asked a month agoDevelopment Endpoint & Glue Version 1.0
asked 3 months agoAWS Glue 3.0 Docker Image - can we increase SPARK configurations ?
asked a month ago
Setting up the VS was easier.
I did not find where to do this step though which is Step 8 on the VS set up
Paste the following JSON and save it.
{ "python.defaultInterpreterPath": "/usr/bin/python3", "python.analysis.extraPaths": [ "/home/glue_user/aws-glue-libs/PyGlue.zip:/home/glue_user/spark/python/lib/py4j-0.10.9-src.zip:/home/glue_user/spark/python/", ] }