AWS ECS CapacityProvider not running tasks

0

Hello,

I have an ECS cluster. That has a capacity provider defined. That capacity provider was set up with an EC2 auto-scaling group that has an AWS ECS AMI. When I add a task to the cluster it recognizes it and the status is set to PROVISIONING. The auto-scaling group then scales up. The capacity provider shows that it scaled up in the Current Size column. But the instances never get listed as part of the cluster and the ECS task never gets assigned.

Does anyone have thoughts on what might be wrong such that the instances are never added to the cluster so the tasks can't utilize the instances?

Thanks

2 Answers
2
Accepted Answer

Hi! At first glance I agree with John, check if your userdata is correct. If you're using Launch Templates, check the userdata at the end in "Advanced Details", it will have to contain this line after the shebang:

echo ECS_CLUSTER=INSERT_YOUR_CLUSTER_NAME >> /etc/ecs/ecs.config;echo ECS_BACKEND_HOST= >> /etc/ecs/ecs.config;

And I would also check some other things:

  • ssh into your new ec2, and check if the contents inside /etc/ecs/ecs/config are correct.
  • Your launch template security groups.
  • The role and policies associated with your instance and services.

Good luck.

answered 2 years ago
  • Thank you! I indeed did not have ECS_CLUSTER set in the config. I added that to the launch template user data so it gets set. I ssh'd into the instance and the configs were correct. I checked the logs (/var/log/ecs/ecs-agent.*) and it mentioned there were no credential providers. My launch template did not have an IAM attached to it so the agent didn't have access to what it needed.

    Adding the IAM to the launch template and setting the ECS_CLUSTER did it.

2

Hello,

From your description, I think the missing piece is some configuration as the nodes of your autoscaling groups start, to join the ECS Cluster: using the ECS optimized AMI means using an image with docker, ecs agent etc, but that doesn't make it automatically join the pool of ECS instances.

If you are using AWS CloudFormation, I would recommend to use user-data + configset (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html) which will allow you to setup packages or render files, etc. If you used something else, then you can only use user-data.

Either way, you need to at least update the value set for ECS_CLUSTER in /etc/ecs/ecs.config (and start/restart the agent) for the node to join the cluster.

Hope this helps

profile picture
answered 2 years ago
  • Thank you! I did not have the cluster set. I also didn't have IAM on the instances configured correctly. See my commend under Tiago-Furlaneto's post for more details.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions