EC2 instance does not show up at ECS container instance

0

I have created ECS cluster with EC2 capacity provider ASG. For other VPCs the created EC2 instance shows up, but just in one VPC it does not show up. So when I check on the EC2 instance for ecs.service logs, it continuously restarted ecs-agent container as follows

Oct 14 00:27:46 ip-172-17-41-185.ap-northeast-2.compute.internal amazon-ecs-init[2781]: level=info time=2024-10-14T00:27:46Z msg="Starting Amazon Elastic Container Service Agent"
Oct 14 00:29:47 ip-172-17-41-185.ap-northeast-2.compute.internal amazon-ecs-init[2781]: level=info time=2024-10-14T00:29:47Z msg="Agent exited with code 1"
Oct 14 00:29:47 ip-172-17-41-185.ap-northeast-2.compute.internal amazon-ecs-init[2781]: level=warn time=2024-10-14T00:29:47Z msg="ECS Agent failed to start, retrying in 16.463436813s"
Oct 14 00:30:04 ip-172-17-41-185.ap-northeast-2.compute.internal amazon-ecs-init[2781]: level=info time=2024-10-14T00:30:04Z msg="Container name: /ecs-agent"
Oct 14 00:30:04 ip-172-17-41-185.ap-northeast-2.compute.internal amazon-ecs-init[2781]: level=info time=2024-10-14T00:30:04Z msg="Removing existing agent container ID: 59d..."
Oct 14 00:30:04 ip-172-17-41-185.ap-northeast-2.compute.internal amazon-ecs-init[2781]: level=info time=2024-10-14T00:30:04Z msg="Starting Amazon Elastic Container Service Agent"
Oct 14 00:32:05 ip-172-17-41-185.ap-northeast-2.compute.internal amazon-ecs-init[2781]: level=info time=2024-10-14T00:32:05Z msg="Agent exited with code 1"
Oct 14 00:32:05 ip-172-17-41-185.ap-northeast-2.compute.internal amazon-ecs-init[2781]: level=warn time=2024-10-14T00:32:05Z msg="ECS Agent failed to start, retrying in 16.02602009s"
Oct 14 00:32:21 ip-172-17-41-185.ap-northeast-2.compute.internal amazon-ecs-init[2781]: level=info time=2024-10-14T00:32:21Z msg="Container name: /ecs-agent"
Oct 14 00:32:21 ip-172-17-41-185.ap-northeast-2.compute.internal amazon-ecs-init[2781]: level=info time=2024-10-14T00:32:21Z msg="Removing existing agent container ID: 2a0..."

What I want to know is the reasons or solutions for the EC2 instance to show up in the container instance in ECS.

  • In addtion as to the ecs.service log, the container fails to start with error "level=error time=2024-10-14T01:25:08Z msg="health check [HEAD http://localhost:51678/v1/metadata] failed with error: Head "http://localhost:51678/v1/metadata\": dial tcp 127.0.0.1:51678: connect: connection refused" module=healthcheck.go"

asked 7 months ago683 views
2 Answers
1

Hello,

You can check container healthcheck config in ecs task definition, for emample: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-task-definition.html

ECS Container HealthCheck

If container healthcheck is HEALTHY, container will be started successfully.

profile pictureAWS
SUPPORT ENGINEER
answered 7 months ago
profile picture
EXPERT
reviewed 7 months ago
0
Accepted Answer

The issue you're experiencing with the EC2 instance not showing up as a container instance in ECS, while it works in other VPCs, could be due to several reasons. Here are some potential causes and solutions:

  1. IAM Role: Ensure that the EC2 instance has the correct IAM role attached. The instance needs the appropriate permissions to communicate with ECS. Verify that the role includes the necessary policies, such as "AmazonEC2ContainerServiceforEC2Role".

  2. Security Groups and Network ACLs: Check if the security groups and network ACLs in the problematic VPC are configured correctly. The EC2 instance needs to be able to communicate with the ECS service endpoints.

  3. ECS Agent Configuration: The ECS agent might not be configured correctly. Check the "/etc/ecs/ecs.config" file on the instance and ensure it contains the correct cluster name and other necessary configurations.

  4. Agent Version: An outdated ECS agent version could cause issues. Try updating the ECS agent to the latest version.

  5. VPC Endpoints: If you're using VPC endpoints, ensure they are set up correctly for the ECS service in the problematic VPC.

  6. Instance Metadata: Make sure the EC2 instance can access the instance metadata service. This is crucial for the ECS agent to function properly.

  7. DNS Resolution: Verify that DNS resolution is working correctly in the VPC.

To troubleshoot further:

  1. Check the full ECS agent logs (usually located at "/var/log/ecs/ecs-agent.log") for more detailed error messages.

  2. Verify the EC2 instance's network connectivity to the ECS service endpoints.

  3. Compare the configurations (IAM roles, security groups, etc.) between the working VPCs and the problematic one to identify any differences.

  4. Try manually stopping and restarting the ECS agent on the instance.

If these steps don't resolve the issue, you may need to terminate the problematic instance and launch a new one with the latest ECS-optimized AMI, ensuring all configurations are correct from the start.
Sources
Updating the Amazon ECS container agent - Amazon Elastic Container Service
Amazon ECS Linux container agent - Amazon Elastic Container Service
Manually updating the Amazon ECS container agent (for non-Amazon ECS-Optimized AMIs) - Amazon Elastic Container Service
Amazon ECS container instance IAM role - Amazon Elastic Container Service

profile picture
answered 7 months ago
profile pictureAWS
EXPERT
reviewed 7 months ago
  • I checked on the security group I missed out the outbound security for EC2

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions