Job failed with status "The job is pending scheduling."

Question

The job is submitted from MWAA airflow. However, the job failed in airflow and EMR Serverless. The status is mentioned as "The job is pending scheduling" in EMR serverless.

Below is the log captured in airflow. Can someone help why the job is failing in EMR serverless while it was in scheduling? , but no log is available in EMR Serverless.

2024-03-04 11:06:24,969] Task failed with exception
Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/utils/waiter_with_logging.py", line 72, in wait
    waiter.wait(**args, WaiterConfig={"MaxAttempts": 1})
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/botocore/waiter.py", line 55, in wait
    Waiter.wait(self, **kwargs)
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/botocore/waiter.py", line 375, in wait
    raise WaiterError(
botocore.exceptions.WaiterError: Waiter serverless_job_completed failed: Waiter encountered a terminal failure state: For expression "jobRun.state" we matched expected path: "FAILED"

Answer

Hello Muthukumar,

Above error stack looks generic. If the EMR-S job failed for any reason, above exception would be captured in the MWAA. Can you please take a look at the failed job in the EMR-S application ? If the job submission fails for any reason, then it should be an issue with spark-submit configuration including the arguments passed. Please validate if they are correct.

If the job failed after created the sparkContext then, the stderr log link would be generated in the job and also make sure the logging is configured to s3.

You can also submit the job directly to EMR-S and test if they are working fine.

Answer

Hi Yogesh,

The job is working fine if submitted manually as well as from airflow. Only in the above scenario, the job submitted from airflow received a job run ID as well, but no status except("the job is pending scheduling") in EMR serverless. both stdout and stderr are not opening.

Job failed with status "The job is pending scheduling."

Relevant content