How do I troubleshoot OutOfMemory errors in Amazon ECS?

Short description

If you experience out of memory issues, then Amazon ECS stops the task and you receive the following error message:

"OutOfMemoryError: Container killed due to memory usage".

This issue occurs when the container's processes use more memory than the amount that you allocated in the task definition. To resolve OutofMemory issues, take the following troubleshooting actions.

Resolution

Identify tasks that Amazon ECS terminated because of memory exhaustion

To identify tasks that Amazon ECS terminated because of memory exhaustion, complete the following steps:

1. Check the stopped tasks on the Amazon ECS console.
2. Look for tasks with 137 or 139 exit codes that typically occur when Amazon ECS experiences memory-related failures.
3. If you find memory issues, then check the container agent logs for heap out of memory issues that caused the task to stop.

Monitor and analyze the container memory usage

To troubleshoot memory-related application issues, use Amazon CloudWatch Logs to analyze your tasks. Use the Amazon ECS or the CloudWatch console to monitor service memory usage. For container-specific memory metrics, set up CloudWatch Container Insights with enhanced observability.

To analyze a container's memory usage over time, use a custom query in CloudWatch Logs Insights.

Example query:

stats max(MemoryUtilized) as mem, max(MemoryReserved ) as memreserved by bin (5m) as period, TaskId, ContainerName
| sort period desc 
| filter ContainerName like "example-container-name" 
| filter TaskId = "example-task-id"

Note: Replace example-container-name with your container name, and example-task-id with your task ID.

Set memory quotas in the Amazon ECS task and container

At the task level, you can set a hard memory quota for the entire task. At the container level, you can use the memoryReservation (memory soft limits) and memory (memory hard limits) parameters to allocate memory.

Before you deploy your application, make sure that your host has enough available memory. Then, update your task definition with soft and hard memory quotas.

Note: You can only use the memoryReservation parameter in Linux containers.

Example task definition:

"containerDefinitions": [
    {
        "name": "example-container-name",
        "memory": 1024,
        "memoryReservation": 512
        // ... other container properties
    }
]

Configure swap for containers that have high transient memory needs

Note: To use the maxSwap and sharedMemorySize parameters, your tasks must use the Amazon Elastic Compute Cloud (Amazon EC2) launch type.

If your containers experience high, temporary memory usage spikes, then update the task definition to use the swap configuration. Swap reduces OutofMemory errors that occur during high load periods. However, swap can also slow your application. After you activate swap, monitor your application's performance.

Adjust your application's memory settings based on your runtime framework

Applications might stop when they reach the framework or runtime default memory quotas, even with available system memory or increased container quotas. To troubleshoot this issue, monitor your application's memory usage patterns. Identify your framework or runtime's default memory settings, and then adjust your task memory parameters to align with your application's requirements.

Java applications

The Java Virtual Machine (JVM) requires an initial memory allocation from the operating system (OS) at startup and has a maximum memory usage quota. For more information, see The java.lang.OutOfMemoryError error on the Oracle website. To adjust the memory values, use JVM flags. For example, run the following command to use -Xms to set the initial heap size and -Xmx to set the maximum heap size:

java -Xms512m -Xmx2048m -jar your-app.jar

To activate detailed garbage collection (GC) logging, add the following JVM argument to your Java command:

-verbose:gc -Xlog:gc*:file=log_filename:time,uptime:filecount=num_files,filesize=file_sizem

For more information, see Garbage Collection logs on the Oracle website.

For advanced memory diagnostics, use jcmd or jmap to generate heap dumps and analysis tools such as JDK Mission Control or VisualVM. For more information, see Diagnostic data and Analysis tools on the Oracle website.

Node.js applications

Use the following v8.getHeapStatistics() module to check the default heap memory quota:

const v8 = require('v8');
console.log(v8.getHeapStatistics());

For more information, see v8.getHeapStatistics() on the Node.js website.

To increase the heap size quota, use the --max-old-space-size parameter when you run your application:

node --max-old-space-size=4096 app.js 

Note: The preceding command sets a 4 gigbyte (GB) heap quota. The default heap memory size can vary across different Node.js versions. For more information, see --max-old-space-size=SIZE (in MiB) on the Node.js website.

To trace garbage collection, add the following flag to the node process:

node --trace-gc server.js

For more information, see Tracing garbage collection on the Node.js website.

For in-depth memory diagnostics, use Node.js V8 built-in profilers, such as --heap-prof, to create heap snapshots. Or, use --heapsnapshot-signal to use Linux signals to create heap snapshots. For more information, see --heap-prof and Using heap snapshot on the Node.js website.

Scale your service to optimize your service-level memory usage

To optimize your service-level memory usage, monitor and adjust your resource allocation to scale tasks based on memory usage patterns. Calculate the total memory that the service tasks use. Then, manually adjust the desired task value in your service based on your memory needs.

Or, use service scaling policies based on memory usage metrics. Configure the scaling policies with thresholds to scale out and scale in.

Avoid future OutofMemory errors

To avoid future OutofMemory errors, perform a load test on the container within a host or server to determine the application's memory needs. Use docker container stats to monitor the containers' memory usage when you're testing. For more information, see docker container stats on the Docker website.

To update your container configuration, take the following actions:

• Manage the data structures.
• Avoid unnecessary object creation.
• Implement caching.
• Use efficient algorithms to reduce memory footprint.

Related information

How do I troubleshoot Amazon ECS tasks that stop or fail to start when my container exits?