Launching the Spark history server and viewing the Spark UI using Docker
Hi,
Have followed the below documentation to set up the Spark History server to see Spark UI Logs. Am able to run the container but not able to access the URL http://localhost:18080 .
docker run -itd -e SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=s3a://<bucket>/logs/ -Dspark.hadoop.fs.s3a.access.key=<ACCESS KEY> -Dspark.hadoop.fs.s3a.secret.key=<Secret Key>" -p 18080:18080 glue/sparkui:latest "/opt/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer"
Where do I check whether the history server has been started and what is the Error info?
- 新しい順
- 投票が多い順
- コメントが多い順
Hello.
Can you check the container status with the following command?
If it has started without any problems, "running" should be displayed.
docker ps -a
You can check the container log with the following command.
Are there any errors shown in the container logs?
docker logs -f container_id
