Skip to content

EMR Virtual cluster unable to delete as Job is stuck. Unable to Cancel Job as it fails and reverts back to Submitted

0

I have an EMR Virtual cluster that ran on an EKS cluster I am unable to delete. The EKS cluster has been deleted and it caused a Job getting stuck in submitted. When I try to cancel the job it goes into Cancel pending, then reverts back to submitted.

Is there anyway to get this job in the failed state so I can delete this EMR Virtual cluster.

Example:

Job IdJob nameStatusCreation time (UTC-05:00)End time (UTC-05:00)Elapsed time
0000000362ssivamgcltestk8jobCancel pendingJuly 30, 2025, 14:52-1 hour, 54 minutes
Job IdJob nameStatusCreation time (UTC-05:00)End time (UTC-05:00)Elapsed time
0000000362ssivamgcltestk8jobSubmittedJuly 30, 2025, 14:52-2 hours, 1 minute

asked a year ago209 views

1 Answer
0

When you have an EMR Virtual cluster that you can't delete because of a stuck job, and the underlying EKS cluster has already been deleted, you're facing a dependency issue where the EMR service is still tracking a job that can't be properly terminated.

Since the job keeps reverting from "Cancel pending" back to "Submitted" state, you'll need to take a different approach to resolve this issue. Here are steps you can try:

  1. First, try to delete the EMR on EKS virtual cluster directly using the AWS CLI command:
aws emr-containers delete-virtual-cluster --id your-virtual-cluster-id

If that doesn't work because of the stuck job, you can try to delete any managed endpoints associated with the virtual cluster:

aws emr-containers delete-managed-endpoint --id managed-endpoint-id --virtual-cluster-id your-virtual-cluster-id

If the job is still preventing deletion, you might need to force the deletion by:

  1. Identifying all virtual clusters in your region:
aws emr-containers list-virtual-clusters --region your-region
  1. Then delete the specific virtual cluster:
aws emr-containers delete-virtual-cluster --region your-region --id your-virtual-cluster-id

Since the underlying EKS cluster is already deleted, the EMR virtual cluster is essentially orphaned. In some cases, you may need to contact AWS Support if none of these methods work, as they might need to help clean up the resources at the service level.
Sources
Use Batch Processing Gateway to automate job management in multi-cluster Amazon EMR on EKS environments | AWS Big Data Blog
Delete an interactive endpoint - Amazon EMR

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.