How do I export a Java heap dump file to an S3 bucket?

Short description

The most common reason why you get an OutOfMemoryError exception is because a Java Virtual Machine (JVM) can't allocate enough memory to a heap. When a Java application runs in an ECS container, a "heap out of memory" error breaks the application and stops ECS tasks. Any .hprof files that the application saves are no longer available. To troubleshoot the exception error and debug memory allocation issues, you must create an .hprof file and save it to an S3 bucket.

Resolution

Note: The following resolution resolves most OutOfMemoryError exception scenarios. For more information about the cause of Java memory errors, see Understand the OutOfMemoryError exception on the Oracle website.

Create an S3 bucket

Use the Amazon S3 console to create an S3 bucket to store your .hprof file.

Create an ECS task definition

Use the Amazon ECS console to create a task definition as a blueprint for your Java application.

For the environment variable, provide the name of your S3 bucket. The application's initialization commands also use the S3 bucket name.

Note: HEAP_DUMP_BUCKET is the environment variable in the example commands.

Create an ECS task IAM role

Create an ECS task AWS Identity and Access Management (IAM) role that includes s3:PutObject permissions so that you can upload files.

Run the initialization commands

In the JVM working directory, a default heap dump is created in a file that's named java_pidpid.hprof. To specify another file name or directory, use the XX:HeapDumpPath= option.

The heap dump is generated in the /tmp/heap_dump.hprof directory in the following example initialization commands:

CMD ["java", \
    "-XX:+HeapDumpOnOutOfMemoryError", \
    "-XX:HeapDumpPath=/tmp/heap_dump.hprof", \
    "-XX:OnOutOfMemoryError=yum install -y aws-cli > /dev/null && aws s3 mv /tmp/heap_dump.hprof s3://$HEAP_DUMP_BUCKET/'date+%F_%T'.hprof ; kill -9 %p", \
    "MemoryApp" \]

Example initialization command flow

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

The OnOutOfMemoryError option specifies a command or script to run when you get an OutofMemoryError exception. For more information about the OnOutOfMemoryError option, see Java HotSpot VM command-line options on the Oracle website.

Complete the following steps for the OnOutOfMemoryError command flow:

1. Install the AWS Command Line Interface (AWS CLI) within -y with no interaction. The command output saves to a temporary file in /dev/null so that unnecessary information about the AWS CLI installation isn't added to the container logs.
2. Upload the .hprof file to the S3 bucket with permissions from the ECS task role. The .hprof file is renamed to follow the date+%F_%T.hprof pattern.
3. Review the Amazon CloudWatch log data:

   2023-05-02T17:02:37.748+01:00    Hello, world! Hello Java! Good bye memory...
   2023-05-02T17:02:37.748+01:00    Going to sleep for 2 ms...
   2023-05-02T17:02:39.748+01:00    Wake now, lets heap the memory...
   2023-05-02T17:02:39.748+01:00    java.lang.OutOfMemoryError: Requested array size exceeds VM limit
   2023-05-02T17:02:39.748+01:00    Dumping heap to /tmp/heap_dump.hprof ...
   2023-05-02T17:02:39.756+01:00    Heap dump file created [2893599 bytes in 0.008 secs]
   2023-05-02T17:02:39.756+01:00    #
   2023-05-02T17:02:39.757+01:00    # java.lang.OutOfMemoryError: Requested array size exceeds VM limit
   2023-05-02T17:02:39.757+01:00    # -XX:OnOutOfMemoryError="yum install -y aws-cli > /dev/null && aws s3 mv /tmp/heap_dump.hprof s3://$HEAP_DUMP_BUCKET/`date +%F_%T`.hprof ; kill -9 %p"
   2023-05-02T17:02:39.757+01:00    # Executing /bin/sh -c "yum install -y aws-cli > /dev/null && aws s3 mv /tmp/heap_dump.hprof s3://$HEAP_DUMP_BUCKET/`date +%F_%T`.hprof "...
   2023-05-02T17:02:54.663+01:00    Completed 256.0 KiB/2.8 MiB (2.2 MiB/s) with 1 file(s) remaining
   Completed 512.0 KiB/2.8 MiB (4.4 MiB/s) with 1 file(s) remaining 
   Completed 768.0 KiB/2.8 MiB (6.6 MiB/s) with 1 file(s) remaining 
   Completed 1.0 MiB/2.8 MiB (8.7 MiB/s) with 1 file(s) remaining 
   Completed 1.2 MiB/2.8 MiB (10.8 MiB/s) with 1 file(s) remaining 
   Completed 1.5 MiB/2.8 MiB (12.9 MiB/s) with 1 file(s) remaining 
   Completed 1.8 MiB/2.8 MiB (15.0 MiB/s) with 1 file(s) remaining 
   Completed 2.0 MiB/2.8 MiB (17.0 MiB/s) with 1 file(s) remaining 
   Completed 2.2 MiB/2.8 MiB (19.1 MiB/s) with 1 file(s) remaining 
   Completed 2.5 MiB/2.8 MiB (21.1 MiB/s) with 1 file(s) remaining 
   Completed 2.8 MiB/2.8 MiB (23.1 MiB/s) with 1 file(s) remaining 
   Completed 2.8 MiB/2.8 MiB (13.0 MiB/s) with 1 file(s) remaining move: ../tmp/heap_dump.hprof to s3://fargate-test-cluster-logs/2023-05-02_16:02:53.hprof
   2023-05-02T17:02:54.705+01:00    Executing /bin/sh -c "kill -9 1"...
   2023-05-02T17:02:54.707+01:00    Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
   2023-05-02T17:02:54.707+01:00    at MemoryApp.main(MemoryApp.java:13)

The preceding log data example uses yum install for the AWS CLI in a container that runs the Amazon Corretto 17 image. If the container image is based on a different operating system (OS), then the command changes. The following are example commands for different operating systems:

• Fedora Linux: dnf install -y awscli
• Debian: apt-get install -y awscli
• Alpine Linux: apk add aws-cli

Note: With any OS, you must use a command that's equivalent to the example yum install command that installs the AWS CLI.