Unexpected ECS error: "CgroupError: Agent could not create task's platform resources"

Question

Since late Dec, my EC2 instance using ECS agent 1.89.2 has been getting the error `CgroupError: Agent could not create task's platform resources` when trying to launch new tasks.

Instances can launch at boot, but eventually the instance "dies" -- it can't launch new tasks. The instance is up and says it's healthy in the EC2 console. Tasks that were launched near instance boot continue to run, but if they crash, new tasks fail to launch with the same error.

I can SSM into the instance and get logs from `/var/log/ecs`, and I see errors like `cgroupv2 create: unable to create v2 manager: dial unix /run/systemd/private: connect: no such file or directory`. Going further, I can see the `/run/systemd/private` socket is present:

```
# stat /run/systemd/private 
  File: /run/systemd/private
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: 17h/23d Inode: 342         Links: 1
Access: (0700/srwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:init_var_run_t:s0
Access: 2024-12-23 04:43:27.589689951 +0000
Modify: 2024-12-23 04:42:47.039999975 +0000
Change: 2024-12-23 04:42:47.039999975 +0000
Birth: 2024-12-23 04:42:47.039999975 +0000
```

I'm not sure if it's the ECS agent, if there's a problem with the AMI, or if there's something else. The AMI name is `al2023-ami-ecs-hvm-2023.0.20241217-kernel-6.1-x86_64`.

My Prod-CA instances are ok, they're running ECS agent 1.89.1, with the AMI name `al2023-ami-ecs-hvm-2023.0.20241213-kernel-6.1-x86_64`. These are the only differences I can definitely see, but I'm not sure I've looked everywhere.

My questions: Is anyone else seeing this? Is a bug, or are we using it wrong? Why did it work pre-Dec-22 (when we switched to the new AMI/agent)?

Henrique Santana · Answer

Hello Dan Turner,

Thank you for bringing this ECS agent issue to our attention. Based on your description and recent reports from other users, there are several potential factors to consider:

1. DOCKER_HOST Configuration: Explicitly specifying DOCKER_HOST in the ECS agent configuration (even with the default value) can cause issues with cgroup management on AL2023. Try the following:

- Remove any DOCKER_HOST specification from /etc/ecs/ecs.config.
    - Ensure it's not set in your user data script either.
    - After removing, restart the ECS agent: sudo systemctl restart ecs

2. Container Image Version: Another customer experiencing a similar issue found that a faulty container image version was the root cause. They resolved the problem by reverting to an older version of their container image. Consider the following:

- Review your task definitions and the container images they use.
    - Try rolling back to a previous version of your container images, especially if the issue started after updating them.
    - Ensure your container images are compatible with the ECS environment and don't have any conflicts with cgroup management.

3. ECS Agent Version: The update from ECS Agent version 1.89.1 to 1.89.2 included an enhancement that [updated the containerd/cgroups library from v3.0.2 to v3.0.4](https://github.com/aws/amazon-ecs-agent/releases/tag/v1.89.2). This change might be contributing to the issue you're experiencing. You could try the following:

- Roll back to ECS Agent version 1.89.1 if it was working correctly for you before.
    - Wait for a newer version of the ECS Agent that might address this issue.

4. AMI Version: If the issue persists, you might want to temporarily roll back to the previous AMI (al2023-ami-ecs-hvm-2023.0.20241213-kernel-6.1-x86_64) that was working correctly for you.

Additional Troubleshooting:

- [ECS Logging collector](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-logs-collector.html) could help you to grab all relevant files to check this further and see if there are additional errors.
- Ensure that your task definitions don't have any configurations that might conflict with cgroup management.

If these steps don't resolve your issue, please provide more details about your ECS task definitions, container images, instance types, and any custom configurations you might be using. This will help in further troubleshooting.

Lastly, it would be helpful to open a support case with AWS if you haven't already. They can provide more specific guidance based on your account and configuration details, and may have additional insights into this issue if it's affecting multiple customers.

re:Post Agent · Answer

Based on the information provided, it appears you're experiencing a specific issue with Amazon ECS (Elastic Container Service) related to cgroup errors when trying to launch new tasks. This problem seems to have started after updating to a newer AMI and ECS agent version.

The error "CgroupError: Agent could not create task's platform resources" suggests that there's an issue with the ECS agent's ability to create and manage cgroups (control groups) for your tasks. The additional error message you're seeing about "cgroupv2 create: unable to create v2 manager" indicates that this is specifically related to cgroup v2.

The fact that tasks can launch at boot but the instance eventually "dies" (unable to launch new tasks) while existing tasks continue to run is interesting. This behavior suggests that there might be a resource leak or a gradual degradation of the ECS agent's ability to manage cgroups over time.

The difference in behavior between your production instances (running ECS agent 1.89.1 and an older AMI) and the problematic instances (running ECS agent 1.89.2 and a newer AMI) suggests that this issue might be related to changes in either the ECS agent or the underlying AMI.

Given that the problem started occurring after switching to the new AMI and agent version, it's possible that there's a compatibility issue or a bug in the newer versions. The error about connecting to the systemd socket (/run/systemd/private) is particularly interesting, as it suggests there might be an issue with how the ECS agent is interacting with systemd.

To address this issue:

1. Consider rolling back to the previous AMI and ECS agent version that were working correctly (1.89.1 and the older AMI) as a temporary measure.

2. Check for any recent updates or known issues with ECS agent version 1.89.2 that might be related to cgroup management or systemd interaction.

3. Verify that your instance has the necessary permissions and configurations to manage cgroups and interact with systemd.

4. Monitor resource usage on the instances to see if there's any correlation between resource consumption and the onset of the issue.

5. Consider reaching out to AWS support for further assistance, as they may have more insight into potential bugs or compatibility issues with the specific AMI and ECS agent versions you're using.

It's possible that this is a bug in the newer ECS agent or AMI, especially given the timing of when the issue started occurring. However, without more widespread reports of similar issues, it's difficult to definitively say if this is a general bug or a specific configuration issue in your environment.  
**Sources**  
[Troubleshooting Amazon ECS ContainerRuntimeError errors - Amazon Elastic Container Service](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-runtime-error.html)  
[Troubleshooting Amazon ECS InternalError errors - Amazon Elastic Container Service](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/internal-error.html)

Unexpected ECS error: "CgroupError: Agent could not create task's platform resources"

Relevant content