How do I troubleshoot my Amazon EMR on EKS jobs that are stuck in SUBMITTED or PENDING state? 

Short description

Use StartJobRun API to submit an Amazon EMR on Amazon EKS job. The job transitions through the PENDING, SUBMITTED, and RUNNING states. If the job stays in the SUBMITTED or PENDING state and fails, then a scheduling issue occurred. The job-submitter pod, job controller, or driver pod wasn't scheduled on the EKS cluster in time.

You might receive one of the following error messages:

• "JobRun timed out before it could move to running state. Neither job controller pod nor driver pod existed when JobRun was marked as timed-out"
• "Pods stuck in Pending or ContainerCreating state"
• "MountVolume.SetUp failed for volume "hadoop-properties": configmap not found"

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Understand the job startup sequence

When you submit an Amazon EMR on EKS job, Amazon EMR on EKS creates a job-submitter pod in your Amazon EKS namespace. The job-submitter pod spawns the Spark driver pod, and then the Spark driver pod creates executor pods.

If step fails, then the job remains stuck in SUBMITTED state until the job-start-timeout parameter expires. The default timeout is 900 seconds, or 15 minutes. To increase the timeout value, update the job-start-timeout value:

{
  "classification": "emr-containers-defaults",
  "properties": {
    "job-start-timeout": "1800"
  }
 }

For more information, see Options for configuring a job run.

Check pod status and events

To identify the pod that doesn't start when the job is stuck, complete the following steps:

1. Run the following command to list all pods in the EMR namespace:

   kubectl get pods -n namespace --sort-by=.metadata.creationTimestamp

   Note: Replace namespace with your Amazon EMR namespace.
2. Run the following command to check pod events for the pending pod:

   kubectl describe pod pod-name -n namespace

   Note: Replace pod-name with your pending pod and namespace with your EMR namespace.
3. Review the kubectl describe pod output.
4. Run the following command to check node availability:

   kubectl get nodes -o wide

Verify cluster resources

Jobs most often stay stuck because the cluster has insufficient resources to schedule the job-submitter or driver pod.

Run the following command to check the current resource utilization:

kubectl top nodes 

Run the following command to check the resource quote configured for a specific namespace:

kubectl get resourcequota -n namespace

Note: Replace namespace with your Amazon EMR namespace.

If you use Karpenter or Cluster Autoscaler, then new nodes might take several minutes to provision.

Run the following command to verify that Karpenter logs don't show throttling errors or internal errors on the CreateFleet API call:

kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter | grep -E "ERROR|error"

Throttling or internal errors on the CreateFleet API call stop Karpenter from provisioning new nodes.

Also, check that you didn't set the Karpenter NodePool expireAfter setting too small. If you set the expireAfter parameter to 10 minutes, nodes terminate before the job starts. Set it to an appropriate value for your workload.

Run the following command to update your Karpenter NodePool expireAfter setting:

kubectl get nodepools -o yaml | grep expireAfter

Note: If you use flexible Amazon Elastic Cloud Compute (Amazon EC2) Spot Instance provisioning, then it's a best practice to configure at least 5 instance type options in your NodePool. Karpenter can provision from a single instance type. However fewer instance type options in your NodePool reduces Spot availability and can increase provisioning delays or interruptions.

Check node taints and tolerations

If all your nodes have taints, the job-submitter pod can't schedule. Pod templates only apply to driver and executor pods, not to the job-submitter pod. If the job-submitter pod can't find a node to schedule, then you receive an error similar to the following:

"0/16 nodes are available: 1 node(s) had untolerated taint {dedicated: production}"

To resolve this issue, maintain at least one untainted general purpose node group on On-Demand instances for the job-submitter pod. For driver and executor pods, make sure that toleration values in pod templates match the taint values of your nodes. If the values don't match between the toleration value and the taint value, then the pod can't schedule.

Note: Pod templates don't apply to job-submitter pods. It's a best practice to keep at least one untainted On-Demand Instance node group for the job-submitter pod. se tainted specialized node groups for driver and executor pods from pod templates.

Configure job-submitter pod placement

To control what node group that the job-submitter pod runs on, use the emr-job-submitter classification in the configurationOverrides parameter of the StartJobRun API with node selectors.

Update the emr-job-submitter classification in your configuration:

{
   "classification": "emr-job-submitter",
   "properties":
 {
     "jobsubmitter.node.selector.eks.amazonaws.com/nodegroup": "general-nodegroup"
   }
 }

Note: Replace general-nodegroup with the name of your node group.

Important: You can't use Pod templates for the job-submitter pod.

For a list of available node selector labels, see Well-Known Labels, Annotations and Taints on the Kubernetes website.

It's a best practice to place job-submitter pods on On-Demand instances. If you place them on Spot instances, then the job might fail when AWS reclaims the instance. You can configure only the fluentd logging container resources. You can't customize resource requests and limits for the job-submitter scheduler container.

Check for missing volumes and ConfigMaps

A pod template can reference a ConfigMap, a Secret, or a PersistentVolumeClaim (PVC). If the driver pod references ConfigMaps, Secrets, or PersistentVolumeClaims that don't exist in the namespace, then the pod stays in Pending or ContainerCreating state. This happens when you create a new virtual cluster or when you didn't correctly configure the Kubernetes RBAC permissions.

To resolve this issue, complete the following steps:

1. Run the following command to check for FailedMount events:

   kubectl describe pod driver-pod -n namespace | grep -A 5 "Events"

   Note: Replace driver-pod with your pending pod and namespace with your EMR namespace.
2. Run the following command to verify ConfigMaps exist:

   kubectl get configmaps -n namespace

   Note: Replace namespace with your EMR namespace.
3. Run the following command to verify PVCs are bound:

   kubectl get pvc -n namespace

   Note: Replace namespace with your EMR namespace.

If you received the "ConfigMap not found" error message and the job worked before without configuration changes, then you didn't correctly configure the AWS Identity and Access Management (IAM) permissions. Or you didn't correctly configure the Kubernetes access entries. Confirm that the Amazon EMR on EKS execution role has the required permissions and that you correctly configured the cluster access entry.

Verify IAM permissions and cluster access

If the job immediately fails and creates no pods, then you have insufficient IAM permissions or Kubernetes RBAC permissions. You receive the following error message:

"JobRun timed out before it could move to running state. Neither job controller pod nor driver pod existed"

To troubleshoot this error, take the following actions:

• Confirm that the IAM role or user submitting jobs has permissions to call emr-containers:StartJobRun.
• Confirm that you correctly configured the EMR on EKS cluster access in the EKS access policy settings.
• Run the following command to confirm that the Kubernetes role bindings for the EMR service account exist in the namespace:

  kubectl get rolebinding -n namespace

  Note: Replace namespace with your EMR namespace.

Check for lingering job-runner pods

If previous job-runner pods weren't cleaned up because of a service-side event or race condition, then they might consume resources or prevent new pods from scheduling. If a regional service event affects EMR on EKS, then the job-runner pods don't clean up.

Run the following command to check for orphaned job-runner pods:

kubectl get pods -n namespace | grep job-runner

Note: Replace namespace with your EMR namespace.

If you find orphaned pods, then check the AWS Service Health Dashboard for active events in your AWS Region. If the job-submitter Kubernetes job is still active, then even if you manually delete job-runner pods, they might be recreated. After the service event resolves, the cleanup automatically happens.

Related information

Using Amazon EMR container defaults classification

Using pod templates

Running Spark jobs with Amazon EMR on EKS

Karpenter on the EMR Containers Best Practices Guides