Why is an Amazon EMR step running even though my application on the YARN completed?

3 minute read
0

An Amazon EMR step is still in the RUNNING state even though the respective Apache Spark or YARN application completed.

Resolution

Use one of the following methods to resolve the issue:

  • Validate the status of respective YARN application and then end the step.
  • Cancel the Step ID manually using the AWS Command Line Interface (AWS CLI).

Validate the status of the YARN application and then end the step

1.    Identify the YARN applicationId from the step logs stderr file. For more information, see How do I troubleshoot a failed step in Amazon EMR?

2.    Connect to the primary node using SSH.

3.    Use the following YARN command to find the state of the YARN application. In the following example command, replace application_id with your application ID. An example application ID is application_1234567891011_001.

yarn application -status application_id

Or, use the following YARN command to list all applications:

yarn application -list -appStates ALL

4.    Check the output of the preceding command for the state of the application.

Application-States: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED]

The following is example output of an application that completed successfully:

Progress : 100%
State : FINISHED
Final-State : SUCCEEDED

If the output of the preceding command has the states FINISHED, FAILED, or KILLED, then the YARN application is completed.

If the application status is NEW, NEW_SAVING, SUBMITTED, ACCEPTED, or RUNNING then the YARN application is still running. Wait for the application to complete or end the application to cancel the step.

7.    Run the following command to end the application. In the following example command, replace application_id with your application ID. An example application ID is application_1234567891011_001.

yarn application -kill application_id

8.    Check the status of the Amazon EMR step again after ending the application.

Cancel the Amazon EMR step manually using the AWS CLI

Note: If you receive errors when running AWS CLI commands make sure that you’re using the most recent version of the AWS CLI.

1.    Use the describe-step command to view the step's status. In the following command, replace cluster-id and step-id with the correct values for your use case.

aws emr describe-step --cluster-id j-xxxxxxxxxxxxx --step-ids s-xxxxxxxx

2.    Use the cancel-steps command to cancel the step. In the following command, replace cluster-id and step-id with the correct values for your use case.

aws emr cancel-steps --cluster-id j-xxxxxxxxxxxxx \
--step-ids s-3M8DXXXXXXXXX \
--step-cancellation-option SEND_INTERRUPT

For more information, see Canceling steps.


AWS OFFICIAL
AWS OFFICIALUpdated 5 months ago