Launching the Spark history server and viewing the Spark UI using Docker

Question

Hi,

Have followed the below documentation to set up the Spark History server to see Spark UI Logs. Am able to run the container but not able to access the URL  http://localhost:18080  .

docker run -itd -e SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=s3a:///logs/ -Dspark.hadoop.fs.s3a.access.key=  -Dspark.hadoop.fs.s3a.secret.key=" -p 18080:18080 glue/sparkui:latest "/opt/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer"

Where do I check whether the history server has been started and what is the Error info?

Answer

Hello.

Can you check the container status with the following command?  
If it has started without any problems, "running" should be displayed.  
```
docker ps -a
```
You can check the container log with the following command.  
Are there any errors shown in the container logs?  
```
docker logs -f container_id
```

Launching the Spark history server and viewing the Spark UI using Docker

관련 콘텐츠