AWS Glue Interactive session

Question

Hi All,

I was following the steps mentioned at below link  to work locally on my system .
https://aws.amazon.com/blogs/big-data/introducing-aws-glue-interactive-sessions-for-jupyter/

This is working absolutely fine . Now I have additional requirement of viewing the jobs **while it is running **in spark UI .
I know we can see job history but I want to see jobs performance while it is running . Assuming localhost:/4040 will provide that , I tried but getting local host not found .

Do I need to install anything extra so that spark ui can provide results on run time ?

Answer

Hello,

I understand you would like to have access to spark ui server to trace the job performance and other details, while running jobs through Glue interactive session.

First, kindly configure the spark ui logs path as shown below before starting the session [1],

%%configure
{
"--enable-spark-ui": "true",
"--spark-event-logs-path": "s3:///sparkui/"
}

Secondly, You can launch the Spark history server using a AWS CloudFormation template that hosts the server on an EC2 instance, or launch locally using Docker. Refer [2] for the steps.

References:
[1] https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-jobs.html
[2] https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-history.html

AWS Glue Interactive session

Relevanter Inhalt