Why is an Amazon EMR step running even though my application on the YARN completed?

Resolution

Use one of the following methods to resolve the issue.

Validate the status of the YARN application and then end the step

Complete the following steps:

1. Identify the YARN applicationId from the step logs stderr file.

2. Connect to the primary node using SSH.

3. Use the following YARN command to find the state of the YARN application. In the following example command, replace application_id with your application ID. An example application ID is application_1234567891011_001.

   yarn application -status application_id

   -Or-

   Use the following YARN command to list all applications:

   yarn application -list -appStates ALL

4. Check the output of the preceding command for the state of the application.
   If the output of the preceding command has the states FINISHED, FAILED, or KILLED, then the YARN application is completed.
   If the application status is NEW, NEW_SAVING, SUBMITTED, ACCEPTED, or RUNNING then the YARN application is still running. Wait for the application to complete or end the application to cancel the step.
   The following is example output of an application that completed successfully:

   Progress : 100%
   State : FINISHED
   Final-State : SUCCEEDED

5. Run the following command to end the application. Replace application_id with your application ID. An example application ID is application_1234567891011_001.

   yarn application -kill application_id

6. Check the status of the Amazon EMR step again after ending the application.

Cancel the Amazon EMR step manually using the AWS Command Line Interface (AWS CLI)

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Complete the following steps:

1. Use the describe-step command to view the step's status. In the following command, replace cluster-id and step-id with the correct values for your use case.

   aws emr describe-step --cluster-id j-xxxxxxxxxxxxx --step-ids s-xxxxxxxx

2. Use the cancel-steps command to cancel the step. In the following command, replace cluster-id and step-id with the correct values for your use case.

   aws emr cancel-steps --cluster-id j-xxxxxxxxxxxxx \
   --step-ids s-3M8DXXXXXXXXX \
   --step-cancellation-option SEND_INTERRUPT

For more information, see Cancel steps when you submit work to an Amazon EMR cluster.