1 Answer
- Newest
- Most votes
- Most comments
1
Hello,
I understand you would like to have access to spark ui server to trace the job performance and other details, while running jobs through Glue interactive session.
First, kindly configure the spark ui logs path as shown below before starting the session [1],
%%configure { "--enable-spark-ui": "true", "--spark-event-logs-path": "s3://<your-bucket>/sparkui/" }
Secondly, You can launch the Spark history server using a AWS CloudFormation template that hosts the server on an EC2 instance, or launch locally using Docker. Refer [2] for the steps.
References: [1] https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-jobs.html [2] https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-history.html
Relevant content
- asked a year ago
- asked 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 years ago