How do I access Spark driver logs on an Amazon EMR cluster?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

On Amazon EMR, Spark runs as a YARN application and supports client and cluster deployment modes.

Client mode jobs

Client mode is the default deployment mode. In client mode, the Spark driver runs on the host where the spark-submit command runs. When you run the spark-submit command with --deploy-mode client on the primary node, you receive the driver logs in the terminal window. Amazon EMR doesn't archive these logs by default. To capture the logs, save the output of the spark-submit command to a file.

Example command:

spark-submit [--deploy-mode client] ... 1>output.log 2>error.log

When you use an Amazon EMR step to submit a Spark application, Amazon EMR archives the driver logs to the stderr.gz file in Amazon Simple Storage Service (Amazon S3). The file path looks similar to the following:

s3://aws-logs-111111111111-us-east-1/elasticmapreduce/j-35PUYZBQVIJNM/steps/s-2M809TD67U2IA/stderr.gz

For more information, see View log files archived to Amazon S3.

To access the Spark driver logs, download the step logs to an Amazon Elastic Compute Cloud (Amazon EC2) instance and then search for warnings and errors.

Complete the following steps:

1. Run the following sync command to download the step logs:

   aws s3 sync s3://aws-logs-111111111111-us-east-1/elasticmapreduce/j-35PUYZBQVIJNM/steps/s-2M809TD67U2IA/ s-2M809TD67U2IA/

2. Run the following command to open the step log folder:

   cd s-2M809TD67U2IA/

3. Run the following command to unzip the log file:

   find . -type f -exec gunzip filename\;

   Note: Replace filename with the name of your file.

4. Run the following command to get the YARN application ID from the cluster mode log:

   grep "Client: Application report for" * | tail -n 1

5. Run the following command to find errors and warnings in the client mode log:

   egrep "WARN|ERROR" *

You can also use applications, such as JupyterHub, Apache Livy, or Apache Zeppelin as a client to submit a Spark application to the cluster. The applications store the driver logs in the application's logs in the /mnt/var/log/ folder on the primary node.

You can also find the compressed logs at the s3://awsexamplebucket/JOBFLOW_ID/node/MASTER_ID/applications/ Amazon S3 path.

If you use Zeppelin, then you can find the Spark driver logs in /mnt/var/log/zeppelin/zeppelin-interpreter-spark-##########.log.

For Jupyter, you can find the driver logs in the /mnt/var/log/livy/livy-livy-server.out Livy logs.

Cluster mode jobs

When you submit the Spark application in cluster mode, the driver process runs in the application primary container. The application primary is the first container that runs when the Spark application runs. The client logs the YARN application report.

To get the driver logs, complete the following steps:

1. Get the application ID from the client logs.
   Example output with the application_1572839353552_0008 application ID.

   19/11/04 05:24:42 INFO Client: Application report for application_1572839353552_0008 (state: ACCEPTED)

2. Identify the application primary container logs.
   Example list of Spark application logs:

   `s3://aws-logs-111111111111-us-east-1/elasticmapreduce/j-35PUYZBQVIJNM/containers/application_1572839353552_0008/container_1572839353552_0008_01_000001/stderr.gz`  
   `s3://aws-logs-111111111111-us-east-1/elasticmapreduce/j-35PUYZBQVIJNM/containers/application_1572839353552_0008/container_1572839353552_0008_01_000001/stdout.gz`  
   `s3://aws-logs-111111111111-us-east-1/elasticmapreduce/j-35PUYZBQVIJNM/containers/application_1572839353552_0008/container_1572839353552_0008_01_000002/stderr.gz`  
   `s3://aws-logs-111111111111-us-east-1/elasticmapreduce/j-35PUYZBQVIJNM/containers/application_1572839353552_0008/container_1572839353552_0008_01_000002/stdout.gz`

   Note: The first container in the list is always the primary container. In the preceding list, the primary container is container_1572839353552_0008_01_000001.

3. Run the following sync command to download the application primary container logs to an instance:

   aws s3 sync s3://aws-logs-111111111111-us-east-1/elasticmapreduce/j-35PUYZBQVIJNM/containers/application_1572839353552_0008/ application_1572839353552_0008/

4. Run the following command to open the Spark application log folder:

   cd application_1572839353552_0008/

5. Run the following command to unzip the log file:

   find . -type f -exec gunzip {} \;

6. Run the following command to search all container logs for errors and warnings:

   egrep -Ril "ERROR|WARN" . | xargs egrep "WARN|ERROR"

7. Run the following command to get the address of the node that the application primary container ran on. On a running cluster, you can use the YARN CLI to get the YARN application container logs:

   yarn logs -applicationId application_1585844683621_0001 | grep  'Container: container_1585844683621_0001_01_000001'

   Expected output:

   20/04/02 19:15:09 INFO client.RMProxy: Connecting to ResourceManager at ip-###-##-##-##.us-west-2.compute.internal/###.##.##.##:8032
   Container: container_1585844683621_0001_01_000001 on ip-###-##-##-##.us-west-2.compute.internal_8041

8. For Spark applications that you submit in cluster mode, run the following command to pull the application primary container logs:

   yarn logs -applicationId application_1585844683621_0001 -containerId container_1585844683621_0001_01_000001 -nodeAddress ip-###-##-##-##.us-west-2.compute.internal

   Note: Replace applicationId with your application's ID, containerId with your container's ID, and nodeAddress ip with your node address.

Related information

Troubleshoot Amazon EMR clusters

Configure Spark