AWS ECS Fargate Container killed by OOM killer without hitting memory container limit

Question

We have an ECS container that is regularly being killed as so:

Exit code: 137
OutOfMemoryError: Container killed due to memory usage

What's confusing is that as far as we can see we are not getting anywhere near its memory usage. We are using DataDog to extract the metrics. At the time that the container is killed these are the values reported for that container:

```
ecs.fargate.mem.task.limit:                     8.00 GiB
ecs.fargate.mem.hierarchical_memory_limit:      7.06 GiB
ecs.fargate.mem.limit:                          7.06 GiB
ecs.fargate.mem.max_usage:                      3.29 GiB
ecs.fargate.mem.usage:                          3.27 GiB
ecs.fargate.mem.cache:                          2.77 GiB
ecs.fargate.mem.mapped_file:                    1.75 GiB
ecs.fargate.mem.rss:                          502.88 MiB
ecs.fargate.mem.inactive_file:                  1.80 MiB
ecs.fargate.mem.active_anon:                    0.00 B
ecs.fargate.mem.active_file:                    0.00 B
```

They've barely changed in the previous 6 hours:

![Lifetime of the container](/media/postImages/original/IMr0WwNLatQzqb9zouUQwApw)

Perhaps I don't understand how to interpret these? My assumption would be that so long as `ecs.fargate.mem.usage` is less than `ecs.fargate.mem.limit` we should be good - and it's more than twice the size.

If it's of interest, the image is built on top of `eclipse-temurin:23.0.2_7-jre-alpine-3.21`.

re:Post Agent · Answer

The situation you're describing is indeed puzzling, as the metrics you've provided suggest that the container is not exceeding its memory limits. However, there are a few potential explanations and things to consider:

1. OOM (Out of Memory) issues can sometimes occur even when the overall memory usage appears to be within limits. This can happen due to memory fragmentation or short-lived spikes in memory usage that aren't captured by the metrics you're seeing.

2. The Java runtime in your container might be experiencing issues with memory management. The JVM's garbage collection process could be struggling to free up memory fast enough, leading to OOM errors even when overall memory usage appears low.

3. There might be a discrepancy between the memory metrics reported by DataDog and the actual memory usage as seen by the container runtime. It's possible that some memory usage is not being accurately captured or reported.

4. The memory limit you're seeing (7.06 GiB) might not be the actual limit enforced by the container runtime. There could be other constraints at the task or host level that are coming into play.

5. Short-lived memory spikes could be occurring between metric collection intervals, causing the OOM killer to act without it being reflected in the metrics you're seeing.

To troubleshoot this issue:

1. Consider increasing the memory limit for your container, even if it appears to be sufficient based on the metrics.

2. Review your Java application's memory settings. You might need to adjust the JVM's heap size or garbage collection parameters.

3. Monitor your application more closely, possibly with more frequent metric collection or by enabling detailed memory profiling within your Java application.

4. Check the ECS task definition to ensure that the memory settings are correctly specified and match what you expect.

5. Review the application logs around the time of the OOM events to see if there are any indications of memory-intensive operations or errors that could be contributing to the problem.

6. Consider using AWS CloudWatch Container Insights for ECS, which might provide additional insights into memory usage patterns that aren't visible in your current metrics.

If the problem persists after these steps, you may need to engage AWS support for a more in-depth investigation of the underlying cause.  
**Sources**  
[Community | Easily Monitor Containerized Applications with Amazon CloudWatch Container Insights](https://community.aws/tutorials/navigating-amazon-eks/eks-monitor-containerized-applications)  
[Troubleshoot status check failures on an unreachable EC2 Linux instance | AWS re:Post](https://repost.aws/knowledge-center/ec2-linux-status-check-failure)  
[Troubleshooting Amazon ECS OutOfMemoryError errors - Amazon Elastic Container Service](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/out-of-memory.html)  
[Diagnose and resolve performance issues in my Amazon ECS-hosted applications | AWS re:Post](https://repost.aws/knowledge-center/ecs-diagnose-performance-issues)

AWS ECS Fargate Container killed by OOM killer without hitting memory container limit

Relevant content