How do I resolve an Amazon EMR cluster that doesn't terminate or terminates earlier than expected when I use an auto-termination policy?

Short description

When you create an Amazon EMR cluster, you can turn on the auto-termination policy. The auto-termination policy terminates the cluster after a specific amount of idle time.

In Amazon EMR versions 5.34 to 5.36 and 6.4.0 or later, a cluster is idle for the following reasons:

• YARN applications aren't active.
• HDFS utilization is below 10%.
• Amazon EMR Notebook or EMR Studio connections aren't active.
• Cluster application user interfaces aren't in use.

In Amazon EMR versions 5.30.0 to 5.33.1 and 6.1.0 to 6.3.0, a cluster is idle for the following reasons:

• YARN applications aren't active.
• HDFS utilization is below 10%.
• Cluster Spark jobs aren't active.

Resolution

To resolve an Amazon EMR cluster that doesn't terminate or terminates earlier than expected when you use an auto-termination policy, complete the following steps:

1. Confirm that the Amazon Elastic Compute Cloud (Amazon EC2) instance profile role EMR_EC2_DefaultRole has the following permissions. If the instance profile role doesn't have the following permissions, then the cluster stays active after it reaches the idle timeout period.

   {
     "Version": "2012-10-17",
     "Statement": {
       "Sid": "AllowAutoTerminationPolicyActions",
       "Effect": "Allow",
       "Action": [
         "elasticmapreduce:PutAutoTerminationPolicy",
         "elasticmapreduce:GetAutoTerminationPolicy",
         "elasticmapreduce:RemoveAutoTerminationPolicy"
       ],
       "Resource": "your-resources"
     }
   }

2. For all Amazon EMR versions, make sure that the metrics-collector process is running. The metrics-collector process collects the metrics to determine auto-termination.

   To check the metrics-collector process, run the following commands:

   ps -ef|grep metrics-collector

   -or-

   systemctl status metricscollector.service

   For more information, see How do I restart a service in Amazon EMR?

3. View the Amazon EMR cluster's CloudWatch metrics. Make sure that the AutoTerminationClusterIdle CloudWatch metric shows 1 in the cluster. If the metric shows 1, then the cluster qualifies for auto-termination. If the metric shows 0, then the cluster is active and doesn't qualify for auto-termination.

4. For Amazon EMR versions earlier than 6.15.0, check whether you overrode the default Java version on the EMR cluster. If you override the default Java version that's used in Amazon EMR, then auto-termination doesn't work as expected.

   To resolve this issue, run the following script on the primary instance of a running cluster or as a bootstrap action on a new cluster:

   #Change EMR daemon startup scripts to use JDK8
   sudo sed -i 's#$JAVA_HOME#/usr/lib/jvm/java-1.8.0-amazon-corretto.x86_64/jre#g' /usr/bin/logpusher
   sudo sed -i 's#$JAVA_HOME#/usr/lib/jvm/java-1.8.0-amazon-corretto.x86_64/jre#g' /usr/bin/instance-controller
   sudo sed -i 's#/usr/bin/java#/usr/lib/jvm/java-1.8.0-amazon-corretto.x86_64/jre/bin/java#g' /usr/bin/metricscollector
   
   #Restart the daemons
   sudo systemctl restart instance-controller
   sudo systemctl restart logpusher
   sudo systemctl restart metricscollector.service