Jupyter hub can't start user server due to botocore.exceptions.NoCredentialsError: Unable to locate credentials

Question

Hello,
I am facing a very frustrating problem, Jupyter Hub is showing up the main page and after logging in with default jovyan/jupyter credentials I cannot start a notebook server.

I setup an EMR cluster with Jupyter Hub, Spark, Livy, Hadoop and Tensorflow. EMR version is 6.10

I update some of the packages by adding a bootstrap.sh file with the following:
```
#!/bin/bash
sudo python3 -m pip install -U setuptools
sudo python3 -m pip install -U pip
sudo python3 -m pip install wheel
sudo python3 -m pip install pillow
sudo python3 -m pip install pandas
sudo python3 -m pip install pyarrow
sudo python3 -m pip install boto3
sudo python3 -m pip install s3fs
sudo python3 -m pip install fsspec
```

the Jupyter configuration in the application config is :
```
[
  {
    "Classification": "jupyter-s3-conf",
    "Properties": {
      "s3.persistence.bucket": "pao-emr-bucket",
      "s3.persistence.enabled": "true"
    }
  }
]
```

The logs i get from the Docker container running jupyter hub are :

```
[I 2023-04-07 16:14:47.900 JupyterHub provider:574] Creating oauth client jupyterhub-user-jovyan
[I 2023-04-07 16:14:47.928 JupyterHub spawner:1483] Spawning jupyterhub-singleuser --port=36543 --debug
[D 2023-04-07 16:14:48.676 SingleUserNotebookApp application:190] Searching ['/home/jovyan/.jupyter', '/home/jovyan/.local/etc/jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2023-04-07 16:14:48.676 SingleUserNotebookApp application:902] Looking for jupyter_config in /etc/jupyter
[D 2023-04-07 16:14:48.676 SingleUserNotebookApp application:902] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2023-04-07 16:14:48.676 SingleUserNotebookApp application:902] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2023-04-07 16:14:48.676 SingleUserNotebookApp application:902] Looking for jupyter_config in /home/jovyan/.local/etc/jupyter
[D 2023-04-07 16:14:48.676 SingleUserNotebookApp application:902] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2023-04-07 16:14:48.677 SingleUserNotebookApp application:902] Looking for jupyter_notebook_config in /etc/jupyter
[D 2023-04-07 16:14:48.798 SingleUserNotebookApp application:923] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
[D 2023-04-07 16:14:48.798 SingleUserNotebookApp application:902] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2023-04-07 16:14:48.799 SingleUserNotebookApp application:902] Looking for jupyter_notebook_config in /opt/conda/etc/jupyter
[D 2023-04-07 16:14:48.799 SingleUserNotebookApp application:902] Looking for jupyter_notebook_config in /home/jovyan/.local/etc/jupyter
[D 2023-04-07 16:14:48.799 SingleUserNotebookApp application:902] Looking for jupyter_notebook_config in /home/jovyan/.jupyter
[D 2023-04-07 16:14:48.799 SingleUserNotebookApp notebookapp:1702] Raising open file limit: soft 1024->4096; hard 65536->65536
[D 2023-04-07 16:14:48.800 SingleUserNotebookApp s3manager:107] s3manager._validate_bucket: User provided bucket: pao-emr-bucket
[I 2023-04-07 16:14:48.891 JupyterHub log:189] 302 GET /hub/spawn/jovyan -> /hub/spawn-pending/jovyan (jovyan@::ffff:172.31.33.43) 1004.81ms
[I 2023-04-07 16:14:48.932 JupyterHub pages:402] jovyan is pending spawn
[D 2023-04-07 16:14:49.838 SingleUserNotebookApp s3_fs:247] S3FS.unprefix: self.prefix_: pao-emr-bucket/jupyter/jovyan path: ['.s3keep']
[D 2023-04-07 16:14:49.838 SingleUserNotebookApp s3_fs:174] S3contents.S3FS: Making dir: `pao-emr-bucket/jupyter/jovyan/.s3keep`
Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 10, in 
    sys.exit(main())
  File "/opt/conda/lib/python3.9/site-packages/jupyter_core/application.py", line 277, in launch_instance
    return super().launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/traitlets/config/application.py", line 1042, in launch_instance
    app.initialize(argv)
  File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/mixins.py", line 852, in initialize
    result = super().initialize(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/mixins.py", line 573, in initialize
    return super().initialize(argv)
  File "/opt/conda/lib/python3.9/site-packages/traitlets/config/application.py", line 113, in inner
    return method(app, *args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/notebook/notebookapp.py", line 2147, in initialize
    self.init_configurables()
  File "/opt/conda/lib/python3.9/site-packages/notebook/notebookapp.py", line 1660, in init_configurables
    self.contents_manager = self.contents_manager_class(
  File "/opt/conda/lib/python3.9/site-packages/s3contents/s3manager.py", line 52, in __init__
    self._fs = S3FS(
  File "/opt/conda/lib/python3.9/site-packages/s3contents/s3_fs.py", line 97, in __init__
    self.init()
  File "/opt/conda/lib/python3.9/site-packages/s3contents/s3_fs.py", line 101, in init
    self.mkdir("")
  File "/opt/conda/lib/python3.9/site-packages/s3contents/s3_fs.py", line 175, in mkdir
    self.fs.touch(path_)
  File "/opt/conda/lib/python3.9/site-packages/s3fs/core.py", line 511, in touch
    self._call_s3(self.s3.put_object, kwargs, Bucket=bucket, Key=key)
  File "/opt/conda/lib/python3.9/site-packages/s3fs/core.py", line 200, in _call_s3
    return method(**additional_kwargs)
  File "/opt/conda/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/conda/lib/python3.9/site-packages/botocore/client.py", line 691, in _make_api_call
    http, parsed_response = self._make_request(
  File "/opt/conda/lib/python3.9/site-packages/botocore/client.py", line 711, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/opt/conda/lib/python3.9/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/opt/conda/lib/python3.9/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/opt/conda/lib/python3.9/site-packages/botocore/endpoint.py", line 115, in create_request
    self._event_emitter.emit(event_name, request=request,
  File "/opt/conda/lib/python3.9/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/opt/conda/lib/python3.9/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/opt/conda/lib/python3.9/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/opt/conda/lib/python3.9/site-packages/botocore/signers.py", line 162, in sign
    auth.add_auth(request)
  File "/opt/conda/lib/python3.9/site-packages/botocore/auth.py", line 373, in add_auth
    raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials
Task exception was never retrieved
future:  exception=HTTPError()>
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/jupyterhub/handlers/base.py", line 996, in spawn_single_user
    await gen.with_timeout(
asyncio.exceptions.TimeoutError: Timeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/jupyterhub/handlers/base.py", line 1030, in spawn_single_user
    raise web.HTTPError(
tornado.web.HTTPError: HTTP 500: Internal Server Error (Spawner failed to start [status=1]. The logs for jovyan may contain details.)
[W 2023-04-07 16:15:16.037 JupyterHub user:767] jovyan's server never showed up at http://127.0.0.1:36543/user/jovyan/ after 30 seconds. Giving up
[E 2023-04-07 16:15:16.066 JupyterHub gen:630] Exception in Future .finish_user_spawn() done, defined at /opt/conda/lib/python3.9/site-packages/jupyterhub/handlers/base.py:900> exception=TimeoutError("Server at http://127.0.0.1:36543/user/jovyan/ didn't respond in 30 seconds")> after timeout
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.9/site-packages/tornado/gen.py", line 625, in error_callback
        future.result()
      File "/opt/conda/lib/python3.9/site-packages/jupyterhub/handlers/base.py", line 907, in finish_user_spawn
        await spawn_future
      File "/opt/conda/lib/python3.9/site-packages/jupyterhub/user.py", line 748, in spawn
        await self._wait_up(spawner)
      File "/opt/conda/lib/python3.9/site-packages/jupyterhub/user.py", line 795, in _wait_up
        raise e
      File "/opt/conda/lib/python3.9/site-packages/jupyterhub/user.py", line 762, in _wait_up
        resp = await server.wait_up(
      File "/opt/conda/lib/python3.9/site-packages/jupyterhub/utils.py", line 236, in wait_for_http_server
        re = await exponential_backoff(
      File "/opt/conda/lib/python3.9/site-packages/jupyterhub/utils.py", line 184, in exponential_backoff
        raise TimeoutError(fail_message)
    TimeoutError: Server at http://127.0.0.1:36543/user/jovyan/ didn't respond in 30 seconds

[I 2023-04-07 16:15:16.067 JupyterHub log:189] 200 GET /hub/api/users/jovyan/server/progress (jovyan@::ffff:172.31.33.43) 27067.75ms
```

Apparently `botocore.exceptions.NoCredentialsError: Unable to locate credentials` is what's making the problem but I cannot get to fix it, could you please help ?

Thank in advance

Answer

Hello rePost-User-0304937, thanks for reaching out!
     
    To address the 'botocore.exceptions.NoCredentialsError: Unable to locate credentials' error. You would need to examine your credential chain. The SDK employs the subsequent sequence to locate credentials:
     
    1.	Environment variables: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
    2.	Web Identity Token credentials: from the environment or container.
    3.	The default credential profiles file: ~/.aws/credentials, which is the most common location, or the config file: ~/.aws/config, generated with the CLI command aws configure.
     
    4.	Amazon ECS container credentials: from the Amazon ECS if the environment variable             AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is set.
    5.	Instance profile credentials: used on EC2 instances and delivered through the Amazon EC2 metadata service.
     
    Make sure your credential do not overlap, as such overlap can lead to an error. 
     
    I hope this helps!
     
    References:
    https://repost.aws/knowledge-center/s3-locate-credentials-error
    https://www.youtube.com/watch?v=UMUQs2PojdE

Jupyter hub can't start user server due to botocore.exceptions.NoCredentialsError: Unable to locate credentials

Relevanter Inhalt