How do I resolve the "java.lang.OutOfMemoryError: GC overhead limit exceeded" error in Amazon EMR?

Short description

The NameNode service uses memory to store namespace objects and metadata for files stored in Hadoop Distributes File System (HDFS). The more files that you have in HDFS, the more memory that NameNode uses.

When the NameNode heap size is insufficient for the amount of HDFS data in the cluster, you receive the "java.lang.OutOfMemoryError: GC overhead limit exceeded" error. To prevent out-of-memory errors, increase the heap size.

Resolution

Check the logs to confirm the error

Complete the following steps:

1. Use SSH to connect to the primary node.

2. To check the status of the NameNode service, run the following command on the primary node:

   initctl list

   When the NameNode service has stopped, you receive the following output:

   hadoop-hdfs-namenode stop/waiting

3. To confirm the OutofMemory error, check the NameNode log at /var/log/hadoop-hdfs/hadoop-hdfs-namenode-ip-####.out.
   Note: Replace #### with the private IP address of the primary node.
   When the NameNode service fails because of an OutofMemory error, you receive an output similar to the following example:

   # java.lang.OutOfMemoryError: GC overhead limit exceeded# -XX:OnOutOfMemoryError="kill -9 %p
   kill -9 %p

Increase the NameNode Heap size

Important: This configuration change requires a restart of the NameNode service. Make sure that no applications perform HDFS read or write operations while you increase the NameNode heap size.

Amazon EMR versions 5.21.0 and later

To increase the heap size, supply a hadoop-env configuration object for the instance group on a running cluster. Or, add the configuration object when you launch a new cluster. Choose a size that's appropriate for your workload.

The following example configuration object increases the heap size from 1 GB to 2 GB:

[
    {
        "Classification": "hadoop-env",
        "Properties": {},
        "Configurations": [
            {
                "Classification": "export",
                "Properties": {
                    "HADOOP_NAMENODE_HEAPSIZE": "2048"
                },
                "Configurations": []
            }
        ]
    }
]

Amazon EMR applies your new configurations and gracefully restarts the NameNode process.

Amazon EMR versions 5.20.0 and earlier

Complete the following steps:

1. Use SSH to connect to the primary node.

2. In the /etc/hadoop/conf/hadoop-env.sh file, increase the NameNode heap size:

   export HADOOP_NAMENODE_HEAPSIZE=2048

   Note: Replace 2048 with your required heap size.

3. Save your changes.

4. To restart the NameNode service, run the following commands:

   sudo stop hadoop-hdfs-namenode; sudo start hadoop-hdfs-namenode

5. To confirm that the NameNode process is running, run the following command:

   initctl list

   Expected output:

   hadoop-hdfs-namenode start/running, process 6324

6. To confirm that HDFS commands work, run the following command:

   hdfs dfs -ls /

   Expected output:

   Found 4 itemsdrwxr-xr-x   - hdfs hadoop          0 2019-09-26 14:02 /apps
   drwxrwxrwt   - hdfs hadoop          0 2019-09-26 14:03 /tmp
   drwxr-xr-x   - hdfs hadoop          0 2019-09-26 14:02 /user
   drwxr-xr-x   - hdfs hadoop          0 2019-09-26 14:02 /var

7. Monitor the cluster after you make changes in configuration.

Related information

Configure applications

How do I resolve "OutOfMemoryError" Hive Java heap space exceptions on Amazon EMR that occur when Hive outputs the query results?