Pod template File not found exception though it is exist

Hello,

There could a couple of reasons as mentioned below that might cause this issue,

1. Try to use local file system directory to pass the pod template and see if that works.
2. Check the execution role arn that submits the spark which has sufficient permission to read the pod file from S3. More details here -

https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/creating-job-execution-role.html &

https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/iam-execution-role.html

3. Check if the mentioned S3 bucket has any bucket policy restriction that might block the read access. https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html
4. You can leverage IAM simulator to test if the given policy can be accessible to s3 pod files - https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_testing-policies.html
5. Test the sample pod template job given this doc which helps to see if the issue with all the pod file template or only with problematic one - https://catalog.us-east-1.prod.workshops.aws/workshops/1f91e1d4-5587-40ff-8d5d-54fc86e0ddc1/en-US/advanced/pod-template
6. Enable and leverage the control plane, audit logs and manager logs to see if anything reported - https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html
7. Run this command before starting their testing. It will capture all events triggered by k8s during their testing. This will help to find if any issue in the events that initiated when launching the spark job.

kubectl get ev -n <namespace used in EMR virtual cluster> -w