By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

ECS EC2 AutoScalingGroup Managed Scaling does not work

0

I am trying to use ECS together with EC2 AutoscalingGroup(ASG) and ManagedScaling.

  • AutoScalingGroupProvider has ManagedScaling ENABLED and TargetCapacity is set to 100.
  • ASG has MinSize=0 and MaxSize=5.

When I create Service in the cluster (service DesiredCount=1), service fails to place the task with following error:

<<serviceName>> was unable to place a task because no container instance met all of its requirements. Reason: No Container Instances were found in your cluster.

I would expect ECS to automatically scale the ASG(add instance) and place the task on the new instance. If I manually change DesiredCapacity of ASG to 1, Service(task) gets deployed. Should not managed scaling take care of it? Can someone guide me on what am I doing wrong? Maybe I need to add some permissions somewhere or I don't know.

NOTE: I am using cloudformation to provision my infrastructure.

Topics
Management & GovernanceComputeContainers
Tags
AWS Auto ScalingAmazon EC2Amazon Elastic Container ServiceAmazon EC2 Auto Scaling
Language
English
simon
asked a year ago579 views
2 Answers
0
Accepted Answer

Hey after consulting with AWS support, the problem was that I was specifying launch type (EC2), but instead I should have specified capacity provider for the service or the entire cluster.

Thanks for the help anyway @Shahad_C

simon
answered a year ago
  • Kyle
    10 months ago

    Hello,

    I am experiencing a similar issue and am wondering if you would be willing to elaborate on your experience? There is no option for the service to use a capacity provider in CloudFormation (EC2 / External / Fargate are the options), and my cluster was set up with an Auto Scaling Group capacity provider. However when I deploy updates via CloudFormation I hit the same error you are receiving and have to manually stop running tasks that need to be replaced. Thanks!

  • simon
    10 months ago

    @Kyle my problem was that I was not specifying the capacity provider on my ECS Service. I had to set CapacityProviderStrategy property (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-service.html#cfn-ecs-service-capacityproviderstrategy)

    and leave the LaunchType property empty...

    Also your capacity provider must be associated with cluster, but I suppose you got that covered.

0

Hello,

In general your understanding is correct. With managed scaling enabled on the capacity provider, an instance should have been launched. Here's some reasons why it might not have happened and things you can check

  1. How long did you wait after seeing the issue before manually changing the Desired? Managed Scaling works by creating a Target Tracking scaling policy, which scales based on reacting to increases in CloudWatch metrics, so the scale-out won't be immediate
  2. Check on the ASG that the Target Tracking scaling policy exists, and see if ECS increased the metric value to 200 after the task placement failed. If the policy doesn't exist, make sure the capacity provider is setup correctly to link it to this ASG
  3. Make sure there are no Suspended Processes on the ASG. If you created it through CFN, there shouldn't be any, but confirm just in case a manual change was made out of band
  4. Check the activity history on the ASG. Had the desired already gone up to 1, and it been trying to launch but getting launch failures?
AWS
Shahad_C
answered a year ago
  • simon
    a year ago
    1. I waited until CircuitBreaker killed my ECS Service creation - this took 20 minutes at least(probably even more). Plenty of time for instance to be spawned. I tried this multiple times.
    2. Check on the ASG that the Target Tracking scaling policy exists, and see if ECS increased the metric value to 200 - tracking policy exists, but when checking the metric in Cloudwatch I do not see any datapoints. When I manually changed desired capacity, metric went to 100%. I also tried ECS Service Rolling deployment(update). And again I would expect a new instance would be spawned(with new task running) but ASG did not scale at all and deployment failed with CircuitBreaker again. Seems like ECS and Service DesiredCount is doing nothing with the group desired capacity.
    3. I do not think so, I do not even know how to check that, but doubt it as it is fresh ASG.
    4. Check the activity history on the ASG. Had the desired already gone up to 1, and it been trying to launch but getting launch failures? History was empty. As I said seems like ECS does not let know that group should scale whatsoever.
  • Shahad_C
    a year ago
    1. Gotcha
    2. If the metric isn't being published, then the alarm won't ever change state, and AutoScaling won't scale. You might want to try deleting and recreating the capacity provider to make sure nothing is messed up on its settings (or the Alarm being created)
    3. Probably not then. FYI, you can see these on the ASG console near the bottom in the 'advanced' section of the main Details tab
    4. Makes sense with the above answers. If the alarm never triggered, the Desired on the ASG wouldn't have changed, so no scaling would have been attempted

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content