Error starting the driver Pod in EKS with spark submit

The error you're encountering suggests that there's an issue with the container image or the command being executed when starting the Spark driver pod. Here are some steps to troubleshoot and potentially resolve the issue:

1. Verify your Docker image: Ensure that your Spark Docker image contains all the necessary components, including the Spark binaries and any required dependencies. The error indicates that the 'driver' executable is not found in the PATH, which could mean the Spark binaries are not properly installed or configured in the image.

2. Check your spark-submit command: Review your spark-submit command to make sure you're specifying the correct main class and jar file. The --class parameter should point to the main class of your Spark application, and the jar file should be accessible within the container.

3. Examine the container image specification: In your spark-submit command, verify that the spark.kubernetes.container.image configuration is correctly pointing to your ECR image. The format should be similar to: --conf spark.kubernetes.container.image=<your-account-id>.dkr.ecr.<region>.amazonaws.com/<your-image-name>:<tag>

4. Check Kubernetes RBAC: Ensure that the service account used by the Spark driver has the necessary permissions to create and manage pods. You mentioned creating a role and role binding, so double-check that these are correctly set up and associated with the service account used in your spark-submit command.

5. Verify the Kubernetes namespace: Make sure you're submitting the Spark job to the correct namespace, and that this namespace is specified in your spark-submit command using the --conf spark.kubernetes.namespace parameter.

6. Review the full pod logs: Try to retrieve and examine the full logs of the driver pod for any additional error messages or clues. You can use the command: kubectl logs <driver-pod-name> -n <namespace>

7. Check node resources: Ensure that your EKS nodes have sufficient resources (CPU and memory) to run the Spark driver pod.

If the issue persists after checking these points, you may need to further investigate your Spark Docker image to ensure it's properly configured for running Spark applications on Kubernetes. Consider using a pre-built EMR on EKS image as a reference or starting point if you're having trouble with a custom image.
Sources
Getting started with spark-submit for Amazon EMR on EKS - Amazon EMR
Error while creating EKS cluster | AWS re:Post
Using YuniKorn as a custom scheduler for Apache Spark on Amazon EMR on EKS - Amazon EMR