pyspark UDF don't execute on EMR jupyterlab

Question

Hello,

I'm facing a pretty annoying error. Whenever I try to execute a UDF function on a EMR Notebook I get the following error:

```
py4j.protocol.Py4JJavaError: An error occurred while calling o157.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 7) (ip-10-192-21-71.ec2.internal executor 1): java.lang.RuntimeException: Failed to run command: /usr/bin/virtualenv -p python3 --no-pip --system-site-packages virtualenv_application_1749834955728_0005_0
	at org.apache.spark.api.python.VirtualEnvFactory.execCommand(VirtualEnvFactory.scala:125)
	at org.apache.spark.api.python.VirtualEnvFactory.setupVirtualEnv(VirtualEnvFactory.scala:83)
	at org.apache.spark.api.python.PythonWorkerFactory.(PythonWorkerFactory.scala:95)
	at org.apache.spark.SparkEnv.$anonfun$createPythonWorker$1(SparkEnv.scala:128)
	at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
	at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:128)
	at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:164)
	at org.apache.spark.sql.execution.python.BatchEvalPythonExec.evaluate(BatchEvalPythonExec.scala:81)
	at org.apache.spark.sql.execution.python.EvalPythonExec.$anonfun$doExecute$2(EvalPythonExec.scala:130)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:138)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1516)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
```

Here's the code:

```
data = [
   Row(zip_code='58542', dma='MIN'),
   Row(zip_code='58701', dma='MIN'),
   Row(zip_code='57632', dma='MIN')
]
df = spark.createDataFrame(data)

@udf(StringType())
def udf_test(dossier):
    return dossier

df = df.withColumn("test", udf_test(df['zip_code']))
df.show()
```

I couldn't find any similar errors on internet besides [this](https://stackoverflow.com/questions/61075287/aws-emr-pyspark-notebook-fails-with-failed-to-run-command-usr-bin-virtualenv) and [this](https://stackoverflow.com/questions/76063408/aws-emr-pyspark-udf-fails-with-failed-to-run-command-usr-bin-virtualenv). The first solution didn't work for me. Setting `"spark.pyspark.virtualenv.enabled": "false"` stops the error but disables install_pypi_package functionality so it's not a good solution for me because I need to install other libraries. I Thought that it could be some problem related to the emr user, so I [enabled user impersonation](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-notebooks-spark-monitor.html) hoping that it could solve the problem, unfortunately it didn't.

I'm running emr-6.9.0 with the following applications:
* JupyterEnterpriseGateway 2.6.0
* Livy 0.7.1
* Spark 3.3.0
* Trino 398

Also, I searched the logs created on S3 but couldn't find any error message that could help me. However, I noticed that hadoop-yarn-timelineserver generated some [exceptions](https://pastebin.com/VDmCMdA0) but I don't know how they are related to UDF execution problem.

Thanks.

guilherme_santos · Answer

After setting
```
%%configure -f
{"conf":{"spark.pyspark.python": "/usr/bin/python3",
         "spark.pyspark.driver.python": "/usr/bin/python3"
        }
}
```
the UDF error disappears but throws the error `RuntimeError: install_pypi_packages can only use called when spark.pyspark.virtualenv.enabled is set to true` when I try to call install_pypi_package. However, after setting 
```
%%configure -f
{"conf":{"spark.pyspark.python": "/usr/bin/python3",
         "spark.pyspark.driver.python": "/usr/bin/python3",
         "spark.pyspark.virtualenv.enabled": "true"
        }
}
```
It throws the error 
```
The code failed because of a fatal error:
	Neither SparkSession nor HiveContext/SqlContext is available..

Some things to try:
a) Make sure Spark has enough available resources for Jupyter to create a Spark context.
b) Contact your Jupyter administrator to make sure the Spark magics library is configured correctly.
c) Restart the kernel.
```
When I try to run any code.

re:Post Agent · Answer

This issue appears to be related to the Python virtual environment configuration in your EMR cluster when executing PySpark UDFs. The error indicates that Spark is trying to create a Python virtual environment but failing when running the command `/usr/bin/virtualenv`.

There are a few approaches you can try to resolve this:

1. **Modify Spark Configuration**: Instead of completely disabling virtual environments (which would prevent you from installing packages), you can try configuring how Spark handles Python environments. Add these configurations when creating your EMR cluster or through the Spark configuration:

```
spark.pyspark.python=/usr/bin/python3
spark.pyspark.driver.python=/usr/bin/python3
```

2. **Use Bootstrap Actions**: You can create a bootstrap action that ensures the virtualenv package is properly installed and configured on all nodes in your cluster.

3. **Alternative UDF Approach**: Try using Spark's pandas UDFs or vectorized UDFs which may handle the Python environment differently.

4. **Check Python Dependencies**: Ensure that all required Python packages for virtualenv are installed on your cluster nodes.

5. **Notebook-scoped Libraries**: If you're using EMR Notebooks, you can try installing notebook-scoped libraries on your running EMR cluster, which might provide a workaround for your package installation needs.

The issue is likely related to how EMR is configured to handle Python environments when executing UDFs across the cluster. The error occurs because Spark is trying to create isolated Python environments on worker nodes to execute your UDF, but the virtualenv command is failing.  
**Sources**  
[Amazon EMR Notebooks - Amazon Web Services](https://aws.amazon.com/emr/features/notebooks/)  
[spark env issues post custom ami creation for EMR on EC2 cluster stuck in steps failure | AWS re:Post](https://repost.aws/questions/QU8rBVh9vgQK-BLj-0pPuONQ/spark-env-issues-post-custom-ami-creation-for-emr-on-ec2-cluster-stuck-in-steps-failure)  
[Amazon EMR Studio | Managed IDE Environment | Amazon Web Services](https://aws.amazon.com/emr/features/studio/)

pyspark UDF don't execute on EMR jupyterlab

Add your answer

Relevant content