Glue Spark History server docker setup - SSL certificate problem

When setting up the Glue Spark History server Docker container in an enterprise environment with SSL certificate restrictions, you're encountering a common issue with self-signed certificates in the certificate chain.

The error you're seeing is related to the Docker build process being unable to download packages from Amazon Linux repositories because it can't validate the SSL certificates. This typically happens in corporate environments where SSL traffic is intercepted by security appliances that use self-signed certificates.

For Amazon Linux 2023 (which appears to be what the Dockerfile is using), the correct command to update certificates would be:

RUN dnf install -y ca-certificates && update-ca-trust

Note that Amazon Linux 2023 uses update-ca-trust rather than update-ca-certificates, and the package manager is dnf rather than apk (which is used in Alpine Linux).

To properly fix this issue, you'll need to:

1. Obtain your organization's root CA certificate(s)
2. Add them to the Docker build context
3. Modify the Dockerfile to copy these certificates into the container
4. Update the certificate trust store

You could modify the Dockerfile with something like:

COPY your-enterprise-ca.crt /etc/pki/ca-trust/source/anchors/
RUN update-ca-trust extract

This would need to be added before any commands that make HTTPS requests.

For a more complete solution, you might need to configure Docker itself to trust your enterprise certificates, as the issue could be occurring at the Docker daemon level before the container build even starts.
Sources
sslv3 alert bad certificate:SSL alert number 42) while SSL handshaking | AWS re:Post
I can't connect although my data source connection options look right (SSL) - Amazon QuickSight
"Certificate is NOT Trusted: self signed certificate" on AWS LightSail | AWS re:Post

Hello, thanks for your question. The error that you are facing when you are installing the Docker image for viewing the Spark UI logs is stemming right in the first line in the Dockerfile:

FROM public.ecr.aws/amazonlinux/amazonlinux:2023

Your docker command is trying to fetch the public ECR image from the official CDN (Content delivery network) for AmazonLinux and the issue is that Docker command was unable to cURL into the URl: "https://cdn.amazonlinux.com/al2023/core/mirrors/2023.7.20250623/x86_64/mirror.list". Since cURL command always checks for SSL connection verification before establishing your connection, the issue is due to your company proxy / VPN that has a different SSL cert-library or ca-certificates, which are your organisational restrictions.

I'm sorry to note that the Github Readme doesn't document the errors that you are facing or HowTos on how to resolve this error, since this error is specific to your company policies and certificates, I would recommend consulting your IT team/ networking team to get this issue resolved. If you are still facing issues, the right platform would be Docker community groups who may help you out even further.

However, if you would like to get the issue resolved at your end, you have two options:

1. Provide the appropriate SSL certs (the method that you are trying)

   But note that the commands you are running are platform specific, hence be advised to run the proper command depending on your platform:

   [1] https://documentation.ubuntu.com/server/how-to/security/install-a-root-ca-certificate-in-the-trust-store/index.html

   [2] https://serverfault.com/questions/1100480/wsl-docker-curl-60-unable-to-get-local-issuer-certificate

2. Skip the SSL cert verification (could be insecure, but if you trust your corporate network): First try the below to see if this works:

    curl --insecure https://cdn.amazonlinux.com/al2023/core/mirrors/2023.7.20250623/x86_64/mirror.list

If it works, you can try to disable it by adding a line in Dockerfile:

ENV CURLOPT_SSL_VERIFYHOST=0
ENV CURLOPT_SSL_VERIFYPEER=0

[3] https://forums.docker.com/t/tls-failed-to-verify-certificate-x509/137486/8

[4] https://stackoverflow.com/questions/54402673/how-to-fix-ssl-certificate-problem-self-signed-certificate-in-certificate-chai

If you would want to just start the Spark history server, you can as well install apache-spark package and java corretto[5] on your machine and manually start the history server without a docker container.

• MacOS: brew install apache-spark
• Linux: dnf/apt install apache-spark

[5] https://aws.amazon.com/corretto/