AWS GLUE Image certificate related issue
Hello Team ,
I have created Docker compose file mentioned below :
version: "2"
services:
spark:
image: glue/spark:latest
container_name: spark
** build: ./spark**
hostname: spark
ports:
- "8888:8888"
- "4040:4040"
entrypoint : sh
command : -c "/home/glue_user/jupyter/jupyter_start.sh"
volumes:
- ../app/territoryhub-replication:/home/glue_user/workspace/jupyter_workspace
Docker file which is getting called in build section is as follows :
FROM amazon/aws-glue-libs:glue_libs_3.0.0_image_01
USER root
RUN mkdir -p /root/.aws
RUN echo "[default]\nregion=us-east-1" >> /root/.aws/config
My Docker is getting started but failing (sharing logs )
Starting Jupyter with SSL
/home/glue_user/jupyter/jupyter_start.sh: line 4: livy-server: command not found
I 2022-05-12 15:41:33.032 ServerApp jupyterlab | extension was successfully linked.
I 2022-05-12 15:41:33.044 ServerApp nbclassic | extension was successfully linked.
I 2022-05-12 15:41:33.046 ServerApp Writing Jupyter server cookie secret to /root/.local/share/jupyter/runtime/jupyter_cookie_secret
I 2022-05-12 15:41:33.541 ServerApp sparkmagic | extension was found and enabled by notebook_shim. Consider moving the extension to Jupyter Server's extension paths.
I 2022-05-12 15:41:33.541 ServerApp sparkmagic | extension was successfully linked.
I 2022-05-12 15:41:33.541 ServerApp notebook_shim | extension was successfully linked.
W 2022-05-12 15:41:33.556 ServerApp All authentication is disabled. Anyone who can connect to this server will be able to run code.
I 2022-05-12 15:41:33.558 ServerApp notebook_shim | extension was successfully loaded.
I 2022-05-12 15:41:33.560 LabApp JupyterLab extension loaded from /usr/local/lib/python3.7/site-packages/jupyterlab
I 2022-05-12 15:41:33.560 LabApp JupyterLab application directory is /usr/local/share/jupyter/lab
I 2022-05-12 15:41:33.565 ServerApp jupyterlab | extension was successfully loaded.
I 2022-05-12 15:41:33.569 ServerApp nbclassic | extension was successfully loaded.
I 2022-05-12 15:41:33.569 ServerApp sparkmagic extension enabled!
I 2022-05-12 15:41:33.569 ServerApp sparkmagic | extension was successfully loaded.
Traceback (most recent call last):
File "/usr/local/bin/jupyter-lab", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/jupyter_server/extension/application.py", line 584, in launch_instance
serverapp = cls.initialize_server(argv=args)
File "/usr/local/lib/python3.7/site-packages/jupyter_server/extension/application.py", line 557, in initialize_server
find_extensions=find_extensions,
File "/usr/local/lib/python3.7/site-packages/traitlets/config/application.py", line 88, in inner
return method(app, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/jupyter_server/serverapp.py", line 2421, in initialize
self.init_httpserver()
File "/usr/local/lib/python3.7/site-packages/jupyter_server/serverapp.py", line 2251, in init_httpserver
max_buffer_size=self.max_buffer_size,
File "/usr/local/lib64/python3.7/site-packages/tornado/util.py", line 288, in new
instance.initialize(*args, **init_kwargs)
File "/usr/local/lib64/python3.7/site-packages/tornado/httpserver.py", line 191, in initialize
read_chunk_size=chunk_size,
File "/usr/local/lib64/python3.7/site-packages/tornado/tcpserver.py", line 134, in init
'certfile "%s" does not exist' % self.ssl_options["certfile"]
ValueError: certfile "/home/glue_user/.certs/my_key_store.pem" does not exist
Please help in resolving this asap
Many Thanks
Hello Team,
For now I have added DISABLE_SSL =TRUE as env variable which makes it work . But if there is better way please let me know if there is better way.
Relevant questions
One-shot docker containers using Docker compose
asked 4 months agoGlue PyShell Python Version
Accepted Answerasked 2 months agoaws-glue-libs:glue_libs_3.0.0_image_01 image issue
Accepted Answerasked a month agoTesting Crawlers with Glue 3.0 Docker image
asked 2 months agoHow do I access the Spark UI?
asked 3 years agoCannot delete the image version after deletion of the pipeline and all related components
asked 2 months agoDocker image outdated
asked 6 months agoAWS GLUE Image certificate related issue
asked 17 days agoAWS Glue Image issue
asked 11 days agoHow to push multi-arch docker image built with docker-compose to ECR
asked 6 months ago