"Task ran out of memory" from containerd but neither the pod or the container were killed.

Question

Hey guys.

So I faced a few cases in the last weeks that the containerd printed "Task ran out of memory" error, but neither the pod nor the container were killed or restarted.

I could see that the Pod memory usage was skyrocketing at that moment, and get stabilized right after the containerd "Task ran out of memory" got printed.

I was expected to see something like: "exit 137" or an increased pod restart counter, but there was nothing.

What is the actual behavior of the containerd when it prints the "Task ran out of memory" error, and what's the condition of the error?

I'm running the EKS v1.27 cluster with amazon-eks-node-1.27-v20230711 AMI (ami-00f80984c1a72a9d1).

Kernel version: 5.10.199-190.747.amzn2.aarch64, Kubelet v1.27.7-eks-e71965b.

Can someone help me here?

Thanks!

Answer

Given the specific versions you provided (EKS v1.27, Kernel version 5.10.199), it may be worth checking for any known issues or updates related to memory management in these versions. could ypu please refer to thoiis eks-ami official documentations for known issues in the provided version :- https://github.com/awslabs/amazon-eks-ami/blob/master/CHANGELOG.md

Answer

if a container exceeds its memory limit, it will be terminated and, depending on the restart policy, the pod might be restarted. This is what typically leads to an "exit 137" status, indicating termination due to memory issue.
Please check  it as below the steps as mention 
kubectl describe pod 
Look at kubelet and containerd logs for more detailed information about the memory handling
 Ensure your EKS cluster, kubelet, and containerd are up-to-date with the latest patches.
Use tools like Prometheus or CloudWatch to monitor the memory usage over time.

"Task ran out of memory" from containerd but neither the pod or the container were killed.

Relevant content