Why does the core node in my Amazon EMR cluster run out of disk space?

Resolution

Determine whether you have unhealthy core nodes

When nodes that have at least one Amazon Elastic Block Store (Amazon EBS) volume attached reach more than 90% disk utilization, Amazon EMR considers these nodes unhealthy.

To determine which nodes have reached 90% disk utilization, complete the following steps: 

1. Check the MRUnhealthyNodes Amazon CloudWatch metric. This metric shows the number of unhealthy nodes in an EMR cluster.
   Note: You can also create a CloudWatch Alarm to monitor the MRUnhealthyNodes metric.
2. Connect to the primary node and access the instance controller log at /emr/instance-controller/log/instance-controller.log.
3. In the instance controller log, search for InstanceJointStatusMap to identify which nodes are unhealthy.
   For more information, see the High disk utilization because of an unhealthy node section in How do I resolve an "ExecutorLostFailure: Slave lost" error in Spark on Amazon EMR?
4. To determine whether a mount has high utilization, log in to the core nodes and then run the following command:

   df -h

Identify what consumes the most amount of disk space 

Identify what files occupy the most space in /mnt/ or /mnt1/ to take appropriate action. The following file types take up the most disk space on Amazon EMR:

• Hadoop Distributed File System (HDFS) data (/mnt/hdfs/)
• YARN container logs (/mnt/var/log/hadoop-yarn/containers/)
• YARN localized files (yarn.nodemanager.local-dirs/filecache, ../usercache/filecache, ../usercache/user/appcache/app-id/)
• Spark application history logs (hdfs:///var/log/spark/apps/)
• A combination of all of the above.

Remove unnecessary local and temporary Spark application files 

When you run Spark jobs, Spark applications create local files that take up the remaining disk space on the core node. For example, if the df -h command shows that /mnt uses more than 90% disk space, then check which directories or files have high utilization.

Run the following command to switch to the /mnt directory:

cd /mnt

Run the following command on the core node to see the top 10 directories that use the most amount of disk space: 

sudo du -hsx * | sort -rh | head -10

If the /mnt/hdfs directory has high utilization, then check the HDFS usage and remove any unnecessary files, such as log files.

To see a comprehensive report of the HDFS cluster status, run the following command:

hdfs dfsadmin -report

To check disk usage for specific direction, run the following command:

hadoop fs -du -s -h /path/to/dir

Note: Replace /path/to/dir with the path to the directory you want to check the space utilization for. 

Reduce the retention period for Spark event and YARN container logs

The /var/log directory stores log files such as Spark event logs and YARN container logs. To clean the log files from HDFS automatically, reduce the retention period.

Reduce the default retention period for Spark job history files

By default, Spark job history files are located in /var/log/spark/apps. When the file system history cleaner runs, Spark deletes job history files older than seven days.

To reduce the default retention period on a running cluster, complete the following steps: 

1. Use SSH to connect to the primary node.

2. Add or update the following values in /etc/spark/conf/spark-defaults.conf:

   ------
   spark.history.fs.cleaner.enabled true 
   spark.history.fs.cleaner.interval 12h 
   spark.history.fs.cleaner.maxAge 1d 
   ------

   The preceding configuration runs the cleaner every 12 hours. This configuration clears files that are more than 1 day old. You can customize this time period in the spark.history.fs.cleaner.interval and spark.history.fs.cleaner.maxAge parameters.

3. Restart the Spark History Server

4. To reduce the default retention period for Spark job history files when you launch the cluster, use the following configuration:

   {
       "Classification": "spark-defaults",
       "Properties": {
           "spark.history.fs.cleaner.enabled": "true",
           "spark.history.fs.cleaner.interval": "12h",
           "spark.history.fs.cleaner.maxAge": "1d";
       }  
   }

You can customize the time period in the spark.history.fs.cleaner.interval and spark.history.fs.cleaner.maxAge parameters.

For more information on these parameters, see Monitoring and instrumentation on the Apache Spark website.

Reduce the default retention period of YARN container logs

Spark application logs are the YARN container logs for your Spark jobs. These logs are located in /var/log/hadoop-yarn/apps on the core node. Spark moves these logs to HDFS when the application finishes its run. By default, YARN keeps application logs on HDFS for 48 hours.

To reduce the retention period, complete the following steps:

1. Use SSH to connect to the primary, core, or task nodes.
2. Open the /etc/hadoop/conf/yarn-site.xml file on each node in your EMR cluster.
3. Reduce the value of the yarn.log-aggregation.retain-seconds property on all nodes.
4. Restart the ResourceManager daemon.

You can also reconfigure the cluster to reduce the retention period.

Configure log rotation for Hadoop and Spark streaming jobs 

A streaming application such as Hadoop or Spark on YARN might write to logs continuously throughout the lifetime of the job. This prevents automatic log cleanup and might lead to disk space filling up over time. 

It's a best practice to implement log rotation for container logs. When you rotate on an hourly basis, you can keep the overall size of each log file down. You can also allow log cleanup mechanisms a chance to upload and delete old logs and save disk space on the node. 

For Spark streaming jobs, configure log rotation in /etc/spark/conf/log4j.properties to rotate logs and use a pattern similar to the following example: 

{[spark.yarn.app](http://spark.yarn.app).container.log.dir}/spark-%d{yyyy-MM-dd-HH-mm-ss}.log

Reduce /mnt/yarn usage

When the disk usage in the /mnt/yarn directory is high, either adjust the user cache retention or scale the EBS volumes on the node. For more information, see How do I stop a Hadoop or Spark job's user cache so that the cache doesn't use too much disk space in Amazon EMR?

Troubleshoot YARN localized files that fill the disk 

If /mnt/yarn/ (yarn.nodemanager.local-dirs) is filling up, it can happen at different stages. 

Intermediate data from MapReduce jobs 

During a MapReduce job, the job writes intermediate data and working files to temporary local storage. Large mapper output can quickly fill disk space.

Resource localization by YARN NodeManager

The NodeManager downloads resources from supported sources to the node's local directory during job execution. 

To resolve the preceding issues, check if running applications and their containers occupy disk space. If no applications are running but the disk remains full of appcache files, then tune the NodeManager deletion service. Adjust NodeManager configuration parameters to trigger the deletion service more quickly.

Note: The NodeManager doesn't delete application cache (appcache) that is actively used by running containers.

The YARN NodeManager provides several properties for managing the localizer cache and file deletion. The yarn.nodemanager.localizer.cache.cleanup.interval-ms property controls how often cache cleanups run. The default is 600000 milliseconds. The yarn.nodemanager.localizer.cache.target**-size-mb** sets the target size of the localizer cache per local directory in MB. The default is 10240. For file deletion, yarn.nodemanager.delete.thread-count determines the number of threads allocated to the deletion service, and yarn.nodemanager.delete.debug-delay-sec specifies how many seconds to wait after an application finishes before its localized files are deleted. The default is 0. 

Resize the cluster or scale Amazon EMR

To avoid HDFS space issues, scale the number of your core nodes. And, if directories other than HDFS directories get full, then scale the number of your core or task nodes. For more information, see Use Amazon EMR cluster scaling to adjust for changing workloads.

You can also extend the EBS volumes in existing nodes or use a dynamic scaling script. For more information, see How do I resolve "no space left on device" stage failures in an Apache Spark job on Amazon EMR?

Related information

Configure Amazon EMR cluster hardware and networking

HDFS configuration

Working with storage and file systems with Amazon EMR

Restart the Spark History Server.