Why can't my Amazon EC2 instance join the Amazon ECS cluster?

Resolution

First, use the AWSSupport-TroubleshootECSContainerInstance AWS Systems Manager runbook to automatically check for potential issues. The AWSSupport-TroubleshootECSContainerInstance runbook automatically troubleshoots common reasons why your EC2 instance can't register or join a cluster.

Note: Make sure that you use the AWSSupport-TroubleshootECSContainerInstance runbook in the same AWS Region as your Amazon ECS cluster and EC2 instance.

If the runbook's output doesn't provide recommendations, then take the following actions to manually troubleshoot the issue.

Prerequisites:

• The EC2 instance must have an AWS Identity and Access Management (IAM) role attached with the necessary permissions to communicate with your Amazon ECS configuration. The IAM role for the instance must also have the ecs:RegisterContainerInstance permissions so that Amazon ECS can register the container instance with the cluster. If you don't need to customize your IAM role, then use the AmazonEC2ContainerServiceforEC2Role.
• Verify that the container instance network configuration is accurate. For more information, see Your Amazon ECS container instance network configuration is incorrect on How can I resolve the "CannotPullContainerError" error in my Amazon ECS EC2 Launch Type Task?

Verify the status of the Amazon ECS agent on the Amazon Linux 2 instance

To check whether the Amazon ECS container agent on the instance is running, run the following command:

sudo systemctl status ecs

If the Amazon ECS container agent isn't running on your instance, then run the following command to start the agent:

sudo systemctl start ecs

The output of the start command looks similar to the following example:

Active: active (running)

To check whether the Docker service is running, run the following command:

sudo systemctl status docker

If the Docker service isn't in a running state, then run the following command to start the service:

sudo systemctl start docker

The output of the start command looks similar to the following example:

Active: active (running)

Check launch configurations

If the launch instance is part of an Auto Scaling group, then verify that the Auto Scaling group's launch configuration is correct. For more information, see the 5. Create a new launch configuration step in Refreshing an Amazon ECS container instance cluster with a new Amazon Machine Image (AMI).

Check the AMI of your container instance

If the container instance's AMI is a copied or custom AMI, then make sure that the instance adheres to the following requirements:

• The Linux distribution runs at least version 3.10 of the Linux kernel.
• You're using the latest version of the Amazon ECS Linux container agent.
• The Docker daemon runs at least version 1.9.0 and all Docker runtime dependencies. For more information, see Install Docker Engine from binaries on the Docker Docs website. To view the current Docker version, run the sudo docker version command. For more information, see Install Docker Engine on the Docker Docs website.

These requirements are preconfigured on Amazon ECS optimized AMIs. It's a best practice to use an Amazon ECS optimized AMI unless your application requires a version that's not available in that AMI. For more information, see Retrieving Amazon ECS optimized Linux AMI metadata.

Check whether the instance's user data contains the correct cluster information

To check whether the instance's user data contains the correct cluster information, run the following command:

#!/bin/bash  echo ECS_CLUSTER=cluster-name >> /etc/ecs/ecs.config

Note: Replace cluster-name with your cluster name.

Verify the log files

If the issue persists, then use Amazon ECS logs collector to collect the logs. Then, review the logs to find the cause. You can also check log files on the container host for the Amazon ECS container agent and Docker.

To view the log files for the Amazon ECS container agent and Docker, run the following commands:

sudo cat /var/log/ecs/ecs-agent.log.YYYY-MM-DD-**sudo cat /var/log/docker

Note: Replace YYYY-MM-DD with the date that you want to check the log files for.

Troubleshoot common errors

Error: Launching a new EC2 instance. Status Reason: This account is currently blocked and not recognized as a valid account. Please contact aws-verification@amazon.com if you have questions. Launching EC2 instance failed.

This error occurs when your AWS account is blocked and Amazon doesn't recognize the account. To unblock your account, send an email to request that AWS unblock your account to aws-verification@amazon.com.

Error: re-registering: ClientException: Container instance 12345678910abcdefghixzy is inactive.\n\tstatus code: 400, request id: 012345678a-012345b-012ab-0a1-9f645f4s5c12" module=agent.go

This error occurs when the Amazon ECS container agent can't use the cluster to register the EC2 container instance because the EC2 instance is inactive. This error is related to the application that runs on the instance. To understand the cause of the error, first check the application. If the error persists, then check the Amazon ECS container agent logs.

Error: Few instances are able to join the cluster but with the same configurations, other instance are not able to join the cluster.

This error occurs when a specific API call exceeds the rate quota and results in a ThrottlingException. For more information, see Request throttling for the Amazon ECS API. To resolve this error, increase the account-level rate quota. Use AWS CloudTrail to check for common API calls such as ListTasks and RegisterContainerInstance.

Error: After changing the instance type, new instances are unable to join the cluster.

This error occurs when the Amazon ECS container agent is stuck in the Pending state and you can't change the instance type. To change the instance type in Amazon ECS, complete the following steps:

1. Delete the container instance.
2. Launch a new container instance that has the new instance type.
   Note: It's a best practice to use an Amazon ECS optimized Amazon Linux 2 AMI to launch the instance for your cluster.

Or, create a new launch configuration, and then update the launch configuration in the Auto Scaling group.

For more information, see How do I change my container instance type in Amazon ECS?

Error: Unable to register as a container instance with ECS: AccessDeniedException: User: arn:aws:sts::1122334455:assumed-role/ecsInstanceRole/i-00aa11bb22cc33def is not authorized to perform: ecs:RegisterContainerInstance on resource: arn:aws:ecs:us-east-1:1122334455:cluster/exampleCluster . status code: 400, request id: 0a123456-7899-10101-a987-6543210deff

-or-

Error: 2019-06-29T16:10:09Z [ERROR] Error re-registering: AccessDeniedException: User: arn:aws:sts::1122334455:assumed-role/ecsInstanceRole/i-0052b2e858b1891ef is not authorized to perform: ecs:RegisterContainerInstance on resource: arn:aws:ecs:us-east-1:1122334455:cluster/exampleCluster status code: 400, request id: 0a123456-7899-10101-a987-123456pqrs

These errors occur when you're missing required IAM permissions. To resolve these errors, create a container instance IAM role. Then, run the AWSSupport-TroubleshootECSContainerInstance runbook to identify the permissions that are missing from the container instance role.

Related information

Why are my Amazon ECS container instances with Amazon Linux 1 AMIs disconnected?

Amazon ECS troubleshooting