AWS ECS CapacityProvider not running tasks
Hello,
I have an ECS cluster. That has a capacity provider defined. That capacity provider was set up with an EC2 auto-scaling group that has an AWS ECS AMI. When I add a task to the cluster it recognizes it and the status is set to PROVISIONING. The auto-scaling group then scales up. The capacity provider shows that it scaled up in the Current Size column. But the instances never get listed as part of the cluster and the ECS task never gets assigned.
Does anyone have thoughts on what might be wrong such that the instances are never added to the cluster so the tasks can't utilize the instances?
Thanks
Hi! At first glance I agree with John, check if your userdata is correct. If you're using Launch Templates, check the userdata at the end in "Advanced Details", it will have to contain this line after the shebang:
echo ECS_CLUSTER=INSERT_YOUR_CLUSTER_NAME >> /etc/ecs/ecs.config;echo ECS_BACKEND_HOST= >> /etc/ecs/ecs.config;
And I would also check some other things:
- ssh into your new ec2, and check if the contents inside /etc/ecs/ecs/config are correct.
- Your launch template security groups.
- The role and policies associated with your instance and services.
Good luck.
Hello,
From your description, I think the missing piece is some configuration as the nodes of your autoscaling groups start, to join the ECS Cluster: using the ECS optimized AMI means using an image with docker, ecs agent etc, but that doesn't make it automatically join the pool of ECS instances.
If you are using AWS CloudFormation, I would recommend to use user-data + configset (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html) which will allow you to setup packages or render files, etc. If you used something else, then you can only use user-data.
Either way, you need to at least update the value set for ECS_CLUSTER in /etc/ecs/ecs.config (and start/restart the agent) for the node to join the cluster.
Hope this helps
Thank you! I did not have the cluster set. I also didn't have IAM on the instances configured correctly. See my commend under
Tiago-Furlaneto's post for more details.
Relevant questions
AWS ECS CapacityProvider not running tasks
Accepted Answerasked 19 days agoECS Capacity Provider Auto-Scaler Instance Selection
Accepted Answerasked 18 days agoALBRequestCountPerTarget auto-scaling metric not available on an ECS Service
asked a month agoECS: Understanding of CapacityProviderReservation
asked 3 months agoECS: Capacity Provider vs Autoscaling Group
asked 7 months agoShould ECS/EC2 ASGProvider Capacity Provider be able to scale-up from zero, 0->1
Accepted Answerasked 6 months agoHow many Load Balancers of what schemes are actually required while creating an ECS cluster with AutoScaling Via Capacity Provider?
asked 5 months agoECS Capacity providers best practices
Accepted Answerasked 2 months agoECS services not scaling in (scale in protection is disabled)
asked 18 days agoUse AWS FARGATE_SPOT as fallback
Accepted Answerasked 2 years ago
Thank you! I indeed did not have
ECS_CLUSTERset in the config. I added that to the launch template user data so it gets set. I ssh'd into the instance and the configs were correct. I checked the logs (/var/log/ecs/ecs-agent.*) and it mentioned there were no credential providers. My launch template did not have an IAM attached to it so the agent didn't have access to what it needed.Adding the IAM to the launch template and setting the
ECS_CLUSTERdid it.