By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

EMR Serverless "Error initializing SparkContext"

0

I'm running into this issue when running a spark job on EMR Serverless.

ERROR SparkContext: Error initializing SparkContext.
java.io.IOException: Mkdirs failed to create /tmp/spark-<uuid>/userFiles-<uuid>/environment

I can't find any useful information on how to troubleshoot this issue. Any ideas on where I can start investigating?

Topics
Analytics
Tags
Amazon EMR Serverless
Language
English
Jareth Rader
asked 2 months ago85 views
1 Answer
1

Do you have the permissions to write to the /tmp directory? Check your IAM. Make sure you have enough disk space too.

Make sure your EMR application is using an instance profile with the right permissions. Check your network config and make sure there are no restrictions accessing the /tmp directory.

Do you have any logs to give yourself more insight?

profile picture
nobledoh
answered 2 months ago
  • Jareth Rader
    2 months ago

    I will check all those things. I pretty much get just one stack trace in the logs as it is failing very early on. But taking another look after you mentioned it, it seems like it might be related to not having enough disk space.

    java.io.IOException: Mkdirs failed to create /tmp/spark-<uuid>/userFiles-<uuid>/environment
	at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:996) ~[hadoop-client-api-3.3.6-amzn-2.jar:?]
	at org.apache.spark.util.Utils$.unpack(Utils.scala:517) ~[spark-core_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
	at org.apache.spark.SparkContext.addFile(SparkContext.scala:1824) ~[spark-core_2.12-3.5.0-amzn-0.jar:3.5.0-amzn-0]
...

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content