EMR on EKS: Executors not created when using pod templates with gp3 storage

Problem Summary

I'm trying to configure Spark executors on EMR on EKS to use additional ephemeral storage instead of filling up the node's /tmp directory. I have two related issues:

1. Pod templates with storage mount: Executors are created successfully, but Spark completely ignores the mounted storage at /opt/spark-local and continues writing to /tmp
2. EMR configuration override: When I try to force Spark to use the mounted storage by setting spark.local.dir, no executors are provisioned at all

Environment

• EMR on EKS: Latest version
• Kubernetes: EKS cluster
• Storage Class: gp3 (confirmed to exist in cluster)
• Node Provisioner: Karpenter

What Works

• Spark jobs run successfully with default storage (no pod templates)
• Driver pods are created successfully
• Executor pods are created successfully when using pod templates with storage

What Doesn't Work

• Issue 1: Spark ignores the mounted storage completely

  • Pod template mounts gp3 storage at /opt/spark-local
  • Environment variables set: SPARK_LOCAL_DIRS=/opt/spark-local, TMPDIR=/opt/spark-local/tmp
  • Storage is mounted and accessible, but Spark continues writing to /tmp (node storage)
  • /opt/spark-local remains empty while /tmp fills up to 80%+ usage

• Issue 2: Setting spark.local.dir prevents executor creation

  • When adding spark.local.dir=/opt/spark-local to EMR configuration
  • No executors are provisioned at all
  • No clear error messages about why executors fail to start

Configuration Details

Pod Template (working for drivers, failing for executors):

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: spark
    spark-role: executor
spec:
  containers:
  - name: spark-kubernetes-executor
    env:
    - name: SPARK_LOCAL_DIRS
      value: "/opt/spark-local"
    - name: TMPDIR
      value: "/opt/spark-local/tmp"
    - name: JAVA_OPTS
      value: "-Djava.io.tmpdir=/opt/spark-local/tmp"
    volumeMounts:
    - name: spark-local-dir
      mountPath: /opt/spark-local
  initContainers:
  - name: volume-permission
    image: busybox
    command: ['sh', '-c']
    args:
    - |
      mkdir -p /opt/spark-local/tmp /opt/spark-local/spark
      chown -R 999:1000 /opt/spark-local
      chmod -R 755 /opt/spark-local
    volumeMounts:
    - name: spark-local-dir
      mountPath: /opt/spark-local
  volumes:
  - name: spark-local-dir
    ephemeral:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          storageClassName: "gp3"
          resources:
            requests:
              storage: 50Gi

EMR Job Configuration:

Configuration 1 (Pod template only - executors created but storage ignored):

'spark.kubernetes.executor.podTemplateFile': 's3://my-bucket/pod-templates/executor_template.yaml'

Configuration 2 (Pod template + spark.local.dir - no executors created):

'spark.kubernetes.executor.podTemplateFile': 's3://my-bucket/pod-templates/executor_template.yaml',
'spark.local.dir': '/opt/spark-local'

Questions

1. Why does Spark ignore the mounted storage and environment variables in the pod template?

   • Storage is successfully mounted at /opt/spark-local
   • Environment variables SPARK_LOCAL_DIRS and TMPDIR are set correctly
   • Yet Spark continues writing to /tmp instead of the mounted volume

2. Why do executors fail to be provisioned when spark.local.dir is set via EMR configuration?

   • Is there a conflict between pod template storage and EMR spark.local.dir setting?
   • What's the correct way to make Spark use custom storage in EMR on EKS?

3. What's the recommended approach for directing Spark temporary storage to mounted volumes in EMR on EKS?

   • Should I use pod templates only?
   • Should I use EMR configuration only?
   • Is there a specific combination that works?

4. Are there EMR-specific restrictions on spark.local.dir that differ from standard Spark on Kubernetes?

5. How can I debug why executors aren't created when spark.local.dir is specified?

What I've Tried

• ✅ Confirmed gp3 storage class exists in the cluster
• ✅ Verified pod template YAML syntax is valid
• ✅ Confirmed storage is successfully mounted at /opt/spark-local in executor pods
• ✅ Verified environment variables (SPARK_LOCAL_DIRS, TMPDIR) are set correctly in containers
• ✅ Tested with different storage classes (gp2, default)
• ✅ Tried hostPath volumes instead of ephemeral volumes
• ✅ Added init containers to set proper permissions (chown 999:1000)
• ✅ Tested with JAVA_OPTS=-Djava.io.tmpdir=/opt/spark-local/tmp
• ❌ Spark continues ignoring mounted storage and uses /tmp
• ❌ Setting spark.local.dir completely prevents executor creation

Expected Behavior

Spark executors should use the mounted gp3 storage at /opt/spark-local for temporary files, shuffle data, and other intermediate storage instead of filling up the node's /tmp directory.

Actual Behavior

• With pod template only: Executors are created successfully but completely ignore the mounted storage, continuing to write everything to /tmp
• With pod template + spark.local.dir: No executors are created at all, job remains in "pending executors" state indefinitely

Any guidance on the correct approach or known limitations would be greatly appreciated!