Why can't my Amazon EC2 instance join the Amazon ECS cluster?

Resolution

To determine why your container instance can't register with your Amazon ECS cluster, run the AWSSupport-TroubleshootECSContainerInstance runbook to automatically check for potential issues.

Run the AWSSupport-TroubleshootECSContainerInstance runbook

Prerequisites:

• The EC2 instance must have an AWS Identity and Access Management (IAM) role attached with the necessary permissions to communicate with your Amazon ECS configuration. The IAM role for the instance must also have the ecs:RegisterContainerInstance permissions so that Amazon ECS can register the container instance with the cluster. If you don't need to customize your IAM role, then use the AmazonEC2ContainerServiceforEC2Role.
• Verify that the container instance network configuration is accurate. For more information, see Your Amazon ECS container instance network configuration is incorrect in How do I resolve "CannotPullContainerError" errors when I launch an EC2 task in Amazon ECS?

To run the runbook, see AWSSupport-TroubleshootECSContainerInstance.

Note: Run the AWSSupport-TroubleshootECSContainerInstance runbook in the same AWS Region as your Amazon ECS cluster and EC2 instance.

Check the runbook's output for suggestions to resolve the issue. If the output doesn't provide recommendations, then take the following actions to manually troubleshoot the issue.

Verify the status of the ECS agent on the container instance

To check whether the Amazon ECS container agent on the instance is running, run the following command:

sudo systemctl status ecs

If the Amazon ECS container agent isn't running on your instance, then run the following command to start the agent:

sudo systemctl start ecs

The output of the start command looks similar to the following:

Active: active (running)

To check whether the Docker service is running, run the following command:

sudo systemctl status docker

If the Docker service isn't in a running state, then run the following command to start the service:

sudo systemctl start docker

The output of the start command looks similar to the following:

Active: active (running)

Check launch configurations

If the launch instance is part of an Auto Scaling group, then verify that the Auto Scaling group's launch configuration is correct. For more information, see the Create a new launch configuration step in Refreshing an Amazon ECS Container Instance Cluster With a New AMI.

Check the AMI of your container instance

If the container instance's AMI is a copied or custom AMI, then confirm that the instance adheres to the following requirements:

• The Linux distribution runs at least version 3.10 of the Linux kernel.
• You're using the latest version of the Amazon ECS Linux container agent.
• The Docker daemon runs at least version 1.9.0 and all Docker runtime dependencies. To view the current Docker version, run the sudo docker version command. For more information, see Install Docker Engine on the Docker Docs website.

These requirements are preconfigured on Amazon ECS-optimized AMIs. It's a best practice to use an Amazon ECS-optimized AMI unless your application requires a version that's not available in that AMI. For more information, see Retrieving Amazon ECS-optimized Linux AMI metadata.

Check whether the instance's user data contains the correct cluster information

To check whether the instance's user data contains the correct cluster information, run the following command:

#!/bin/bash
echo ECS_CLUSTER=cluster-name >> /etc/ecs/ecs.config

Note: Replace cluster-name with your cluster name.

Verify the log files

If the issue persists, then use the Amazon ECS logs collector to collect the logs. Then, review the logs to find the cause. You can also check log files on the container host for the Amazon ECS container agent and Docker.

To view the log files for the Amazon ECS container agent and Docker, run the following commands:

sudo cat /var/log/ecs/ecs-agent.log.YYYY-MM-DD-##

sudo cat /var/log/docker

Note: Replace YYYY-MM-DD with the date that you want to check the log files for.

Troubleshoot errors

Error: Launching a new EC2 instance. Status Reason: This account is currently blocked and not recognized as a valid account. Please contact aws-verification@amazon.com if you have questions. Launching EC2 instance failed.

This error occurs when your AWS account is blocked and Amazon doesn't recognize the account. To unblock your account, send an email to aws-verification@amazon.com to request that AWS unblock your account.

Error: re-registering: ClientException: Container instance 12345678910abcdefghixzy is inactive. status code: 400, request id: 012345678a-012345b-012ab-0a1-9f645f4s5c12 module=agent.go

This error occurs when the Amazon ECS container agent can't use the cluster to register the EC2 container instance because the EC2 instance is inactive. This error is related to the application that runs on the instance. To understand the cause of the error, first check the application. If the error persists, then check the Amazon ECS container agent logs.

Error: Few instances are able to join the cluster but with the same configurations, other instances are not able to join the cluster.

This error occurs when a specific API call exceeds the rate quota and results in a ThrottlingException. For more information, see Request throttling for the Amazon ECS API. To resolve this error, increase the account-level rate quota. Use AWS CloudTrail to check for API calls such as ListTasks and RegisterContainerInstance.

Error: After changing the instance type, new instances are unable to join the cluster.

This error occurs when the Amazon ECS container agent is stuck in the Pending state and you can't change the instance type. To change the instance type in Amazon ECS, complete the following steps:

1. Delete the container instance.
2. Launch a new container instance that has the new instance type.

Note: It's a best practice to use an Amazon ECS-optimized Amazon Linux 2023 AMI to launch the instance for your cluster.

Or, create a new launch configuration, and then update the launch configuration in the Auto Scaling group.

For more information, see How do I change my container instance type in Amazon ECS?

Error: Unable to register as a container instance with ECS: AccessDeniedException: User: arn:aws:sts::1122334455:assumed-role/ecsInstanceRole/i-00aa11bb22cc33def is not authorized to perform: ecs:RegisterContainerInstance on resource: arn:aws:ecs:us-east-1:1122334455:cluster/exampleCluster. status code: 400, request id: 0a123456-7899-10101-a987-6543210deff

This error occurs when you're missing required IAM permissions. To resolve this error, create a container instance IAM role. Then, run the AWSSupport-TroubleshootECSContainerInstance runbook to identify the permissions that are missing from the container instance role.

Related information

Amazon ECS troubleshooting

Amazon ECS container instance IAM role

AWSSupport-TroubleshootECSContainerInstance

Why are my Amazon ECS container instances with Amazon Linux 1 AMIs disconnected?