By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

How do I troubleshoot a disconnected Amazon ECS agent?

7 minute read
1

My container instances for Amazon Elastic Container Service (Amazon ECS) are disconnected.

Short description

It's normal for your Amazon ECS container agent to disconnect and reconnect multiple times in an hour as part of the normal operation. These change events aren't a cause for concern. Connection events that last for only a few minutes might not indicate issues with the container agent or your container instance.

However, if the container agent remains in the disconnected state for longer, then the container instance can't operate as part of your Amazon ECS cluster. This issue might be caused by one of the following reasons:

  • Networking issues prevent communication between the instance and Amazon ECS.
  • The container agent doesn't have the required AWS Identity and Access Management (IAM) permissions to communicate with Amazon ECS endpoints.
  • There are problems with the host or Docker daemon inside the container instance.
  • There's resource contention in the underlying host.

Note: It's a best practice to use the latest version of Amazon ECS container agent.

Resolution

Note: The following resolution applies to Amazon ECS-optimized Amazon Linux 2 AMIs.

You can use SSH keys to connect to your Amazon EC2 instances. If you don't have the SSH keys generated, then you can use Session Manager, a capability of AWS Systems Manager, to connect to your instance. By default, Systems Manager Agent is installed on Amazon Linux 2 AMIs and Amazon Linux 2 ECS-optimized base AMI.

Verify that the container agent is running on the container instance

To verify the status and connectivity of the Amazon ECS container agent, run one of the following commands on your container instance:

$ sudo systemctl status ecs            
$ sudo docker ps -f name=ecs-agent

The output specifies active (running) and looks similar to the following:

ecs.service - Amazon Elastic Container Service - container agent   Loaded: loaded (/usr/lib/systemd/system/ecs.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-02-15 15:51:09 UTC; 37min ago
     Docs: https://aws.amazon.com/documentation/ecs/
  Process: 30039 ExecStopPost=/usr/libexec/amazon-ecs-init post-stop (code=exited, status=0/SUCCESS)
  Process: 29987 ExecStop=/usr/libexec/amazon-ecs-init stop (code=exited, status=0/SUCCESS)
  Process: 30077 ExecStartPre=/usr/libexec/amazon-ecs-init pre-start (code=exited, status=0/SUCCESS)
 Main PID: 30123 (amazon-ecs-init)
    Tasks: 5
   Memory: 3.7M
   CGroup: /system.slice/ecs.service
           └─30123 /usr/libexec/amazon-ecs-init start

CONTAINER ID   IMAGE                                            COMMAND    CREATED      STATUS                PORTS     NAMES
eb1dc8d4ab3b   amazon/amazon-ecs-agent:latest   "/agent"        3 days ago   Up 3 days (healthy)                        ecs-agent

If the issue is caused because of a disconnected agent, then run the following command to restart the ECS agent:

$ sudo systemctl restart ecs

Note: There's no output returned after you run the preceding commands.

To verify that the agent is running, run the following command:

sudo systemctl status ecs

Verify that the Docker service is running on the container instance

To verify that the Docker service is running on the affected container instance, run the following command:

sudo systemctl status docker

The output specifies active (running) and looks similar to the following:

docker.service - Docker Application Container Engine   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2022-02-11 17:42:32 UTC; 3 days ago
     Docs: https://docs.docker.com
  Process: 4307 ExecStartPre=/usr/libexec/docker/docker-setup-runtimes.sh (code=exited, status=0/SUCCESS)
  Process: 4296 ExecStartPre=/bin/mkdir -p /run/docker (code=exited, status=0/SUCCESS)
 Main PID: 4315 (dockerd)
    Tasks: 24
   Memory: 360.5M
   CGroup: /system.slice/docker.service
           ├─4315 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --default-ulimit nofile=32768:65536
           ├─6010 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.17.0.2 -container-port 80
           └─6016 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 80 -container-ip 172.17.0.2 -container-port 80

If the Docker service is inactive, then run the following command to restart the Docker service:

sudo systemctl restart docker

Note: There's no output returned after you run the preceding commands.

To verify that the Docker service has restarted, run the following command:

sudo systemctl status docker

Review log files for the container agent and Docker

If your container instance is still disconnected, then review the log files on the container host for the container agent and Docker.

Check the following log files for keywords, such as "error", "warn", or "agent transition state":

  • View the Amazon ECS container agent's latest logs at /var/log/ecs/ecs-agent.log
    Note:     You can view the rotated log by filtering to /var/log/ecs/ecs-agent-log.timestamp
  • View the Amazon ECS init log at /var/log/ecs/ecs-init.log
  • View the userdata execution logs at /var/log/cloud-init.log
  • View the Docker Daemon logs with the command sudo journalctl -u docker

If you use Linux, then you can also review the exit code for more information on the stopped agent container. For more information, see exitcodes on the GitHub website.

To get the exit code, run the following command:

docker inspect your container ID

Replace your container ID with the ID of the stopped container.

Note: You can use the Amazon ECS logs collector to collect general operating system logs, Docker logs, and container agent logs for Amazon ECS.

Verify that the IAM instance profile has the necessary permissions

If the container agent is still disconnected, verify that the IAM instance profile that's associated with the container instance has the necessary IAM permissions:

  1. Use SSH or Session Manager to connect to the instance.

  2. To view the instance metadata on the instance profile associated with the instance, run the following command:

    curl http://169.254.169.254/latest/meta-data/iam/info

    The output looks similar to the following:

    {
  "Code" : "Success",
  "LastUpdated" : "2022-02-16T22:42:17Z",
  "InstanceProfileArn" : "arn:aws:iam::1122334455:instance-profile/ecsInstanceRole",
  "InstanceProfileId" : "AIPA4VIZXOFF55F72XIZN"
}

  3. Verify that the IAM role contains the correct permissions for your container instances.

  4. To verify specific credential errors, run a command similar to the following to check the container agent log for a list of ECS logs:

    cat /var/log/ecs/ecs-agent.log.YYYY-MM-DD-##

    Replace YYYY-MM-DD-## with the relevant timestamp.

Note: The container agent log is rotated every hour. The suffix automatically changes to reflect the current date and time. Update the command to include the date range and log ID for when the issue occurred.

Verify that your container instance has enough resources to run the ECS agent

If your tasks have a high memory/CPU utilization, then your container instance might not have enough resources to run the ECS agent.

The Amazon ECS container agent uses the Docker ReadMemInfo() function to query the amount of memory that's available for the operating system.

Run the following command on your container instance to view the total memory that's recognized by the operating system:

free -b

Example output for an t2.large instance running the Amazon ECS-optimized Amazon Linux AMI:

                          total        used         free                   shared     buff/cache    available
Mem:                    8361193472   298577920     7325388800              405504      737226752    7844274176
Swap:                     0              0           0

You can choose to reserve some memory for the Amazon ECS container agent and other critical system processes on your container instances. Reserving this memory helps to make sure that your task's containers don't contend for the same memory. For more information, see Reserving Amazon ECS Linux container instance memory.

Verify that the environment variable ECS_CLUSTER has the correct cluster name

If the Amazon ECS container agent configuration parameter ECS_CLUSTER has the incorrect cluster name, then the container instance can't join the cluster. To check the contents of the /etc/ecs/ecs.config file to verify this parameter, run the following command:

cat /etc/ecs/ecs.config

Verify that the ECS agent can communicate to ECS endpoints

To connect with ECS endpoints, the network access control lists and container instance security group must allow outbound connections on port 443 (HTTPS).

To check the outbound connections to ECS endpoints (ACS/TCS), run one of the following commands on your container instance:

sudo yum install telnet -y$ telnet ecs.region.amazonaws.com 443

-or-

$ curl https://ecs.region.amazonaws.com

Review the following best practices:

Related information

Amazon ECS troubleshooting

Create the container instance role

Viewing Amazon ECS container agent logs

Topics
Containers
Tags
Amazon Elastic Container Service
Language
English
AWS OFFICIAL
AWS OFFICIALUpdated 4 months ago
No comments

Relevant content