ECS Agent frequently loses connection with the ECS service and drops out of the ECS cluster

This issue occurs when the Amazon ECS agent disconnects from the ECS service. The error message indicates that the container instance is considered "missing" by the ECS service, which is why it's dropping out of the cluster.

To troubleshoot and potentially resolve this issue, you can try the following approaches:

1. Restart the ECS agent on the affected instances. For Amazon Linux 2 instances, you can use:

sudo systemctl restart ecs

For Amazon Linux 1 instances:

sudo stop ecs && sudo start ecs

2. Check the ECS agent logs for more detailed information about the disconnection. The logs are typically located at /var/log/ecs/ecs-agent.log. Look for any patterns or specific errors that might be causing the disconnection.

3. Verify network connectivity between your EC2 instances and the ECS service. Network interruptions, even brief ones, can cause the agent to disconnect.

4. Ensure your instances have sufficient resources (CPU, memory) to run the ECS agent properly. Resource constraints can sometimes cause the agent to become unresponsive.

5. Check if there are any security group or network ACL settings that might be interfering with the connection between the ECS agent and the ECS service.

6. Consider implementing auto-recovery for your instances to automatically recover them if they become impaired due to underlying hardware issues.

7. If you're using custom AMIs, ensure they have the latest version of the ECS agent installed.

If the issue persists, you might want to consider implementing a more robust solution than just terminating disconnected instances. For example, you could create a script that attempts to restart the ECS agent first before deciding to terminate the instance.
Sources
Troubleshoot schedule issues for Amazon ECS tasks on EC2 instances | AWS re:Post
Amazon ECS API failure reasons - Amazon Elastic Container Service
Amazon ECS service event messages - Amazon Elastic Container Service
Connectivity Issue to EC2 Instances | AWS re:Post

It's normal for your Amazon ECS container agent to disconnect and reconnect multiple times in an hour as part of the normal operation. These change events aren't a cause for concern. Connection events that last for only a few minutes might not indicate issues with the container agent or your container instance. Please refer to this documentation[1] for the same.

However, if the container agent remains in the disconnected state for longer, then the container instance can't operate as part of your Amazon ECS cluster. This issue might be caused by various factors.

If the agent is in disconnected state for a longer duration of time then you can follow the below recommendations:

1. Run the following command to restart the ECS agent: For Amazon ECS-optimized Amazon Linux 2 AMIs:

$ sudo systemctl restart ecs

To verify that the agent is running, run the following command:

$ sudo systemctl status ecs

2. Update the ECS Agent and underlying AMI [2]: It's a best practice to use the latest version of Amazon ECS container agent.

3. Review the log files on the container host for the container agent and Docker. Check the log files for keywords, such as "error", "warn", or "agent transition state". View the Amazon ECS container agent's latest logs at /var/log/ecs/ecs-agent.log

4. Verify IAM permissions: Verify that the IAM role attached to your EC2 instances has the necessary permissions, confirm if it has the AmazonEC2ContainerServiceforEC2Role managed policy [3]. The ECS agent needs to register and maintain its connection with the control plane, and a misconfigured IAM role can cause the service to reject the instance's registration.

5. Check for resource contention: Ensure your instances have sufficient resources (CPU, memory) to run the ECS agent properly. High resource usage on an instance can cause the ECS agent to become unresponsive, leading to a dropped connection.

As best practice, you can reserve some memory for the Amazon ECS container agent and other critical system processes on your container instances [4]. Reserving this memory helps to make sure that your task's containers don't contend for the same memory.

6. Mention VPC endpoints: If the instances are in a private subnet with no internet access, a VPC endpoint for ECS is necessary for the agent to communicate with the ECS service without going over the public internet.

Also, verify the container instance security groups along with NACL's and make sure they aren't restrictive in nature that is, the security groups should not block traffic towards the endpoint.

7. Enable debug logging, to check why the connection was lost.

In order to collect the entire details to debug the issue, I recommend you to leverage the ECS log collector script [5] Additionally, for more detailed analysis of the issue, you can consider reaching out to AWS Premium Support [6]

Sources: [1] https://repost.aws/knowledge-center/ecs-agent-disconnected-linux2-ami

[2] https://github.com/aws/amazon-ecs-ami/releases

[3] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/instance_IAM_role.html

[4] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/memory-management.html#ecs-reserved-memory

[5] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-logs-collector.html

[6] https://aws.amazon.com/premiumsupport/