ECS cluster doesn't startup another EC2 instance when deploying

Question

I have a task I want to run, but deployment of that task when I try to update a running task never deploys. I have a cluster that is running a single EC2 node. When I perform an update service and force a new deployment that deployment sits idle waiting for that one instance to free up. So in order to get it to deploy I have stop the existing task, wait for it to drain, deprovision, shutdown the tasks, etc. Eventually, the new task runs, but only after the existing task stops. It takes quite a long time to go through that cycle. It's adding too much down time to be usable in production.

What I want is when a new deployment starts, the auto scaling group will start up a new EC2 instance, ECS will deploy the new task to that instance, then stop the old task, deprovision and shutdown the old EC2 instance. I have my deployment configuration is the following:

## Deployment Configuration
Deployment controller type: ECS Deployment strategy: Rolling update Deployment circuit breaker: Turned on Min and max running tasks: 100% min and 200% max

## Auto Scaling Group Configuration
Desired capacity: 1 Scaling limits (Min - Max): 1 - 2 Desired capacity type: Units (number of instances)

## Dynamic Scaling Policy
Policy type: Target tracking scaling Enabled or disabled: Enabled Execute policy when: As required to maintain Custom CloudWatch metric at 100 Take the action: Add or remove capacity units as required Instances need: 300 seconds to warm up before including in metric Scale in: Enabled

## CloudWatch Metric:
This is the metric used for autoscaling on the autoscaling group:
```
{
   "CustomizedMetricSpecification": {
    "MetricName": "CapacityProviderReservation",
    "Namespace": "AWS/ECS/ManagedScaling",
    "Dimensions": [
      {
        "Name": "CapacityProviderName",
        "Value": "Infra-ECS-Cluster-productioncluster-7854937b-AsgCapacityProvider-9ghdkeune74"
      },
      {
        "Name": "ClusterName",
        "Value": "production_cluster"
      }
    ],
    "Statistic": "Average"
  }
}
```
Why isn't the auto scaling group resizing the instances to handle deployments?

re:Post Agent · Answer

The issue you're experiencing is related to how ECS manages task placement and capacity during deployments. Even though you've configured your deployment settings with 100% minimum and 200% maximum running tasks, which should theoretically allow for two tasks to run simultaneously during deployment, the auto scaling group isn't scaling up as expected.

Here are the likely reasons and solutions:

1. **Capacity Provider Reservation Metric**: The CapacityProviderReservation metric you're using tracks the percentage of resources that are being used by ECS tasks. During a deployment, ECS first tries to place the new task on existing instances before triggering a scale-out event. If your existing instance doesn't have enough capacity to run both tasks simultaneously (due to CPU, memory, or port conflicts), the new task will remain in a pending state, but this might not immediately trigger scaling.

2. **Port Conflicts**: If your container is using host networking mode and binding to specific ports, the new task can't start because the required port is already in use by the existing task. This is a common issue when using the 'host' networking mode.

3. **Warm-up Time**: Your configuration shows a 300-second (5-minute) warm-up time for new instances. This means that even if a scale-out event is triggered, there will be a delay before the new instance is considered ready.

To address these issues:

1. **Consider using awsvpc network mode** instead of host networking to avoid port conflicts. This allows multiple tasks to run on the same instance without port conflicts.

2. **Adjust your capacity provider strategy** to be more aggressive about scaling out. You might need to lower the target value from 100% to something like 70-80% to trigger scale-out events earlier.

3. **Review your task placement strategies and constraints** to ensure they're not preventing the new task from starting.

4. **Enable managed instance draining** in your capacity provider settings. This allows ECS to gracefully remove tasks from an instance that's being scaled in.

5. **Implement a rolling update strategy** with appropriate minimum and maximum percentages to gradually replace old tasks with new ones.

By implementing these changes, you should be able to create a more efficient deployment process that scales out when needed for deployments and scales back in when the extra capacity is no longer required.  
**Sources**  
[Running Update Service shows a pending task, but never shuts down the existing task so a new deployment runs | AWS re:Post](https://repost.aws/questions/QUsDLK7QhMRFK0L_2TeSBF_Q/running-update-service-shows-a-pending-task-but-never-shuts-down-the-existing-task-so-a-new-deployment-runs)  
[ECS Task Deployment Doesn't Reserve More Capacity | AWS re:Post](https://repost.aws/questions/QUWg4dOlvpQkK_lAwMR5Qo8A/ecs-task-deployment-doesn-t-reserve-more-capacity)

Riku_Kobayashi · Answer

Hello.

What is your ECS task placement strategy configured?  
I thought it might be possible that "binpack" was configured to minimize EC2 instances.  
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-placement-strategies.html

I have also experienced situations in the past where AutoScaling did not work properly because EC2 could not be launched due to quota restrictions on EC2 instance types.  
So, please check if any of the above is happening.

ECS cluster doesn't startup another EC2 instance when deploying

Deployment Configuration

Auto Scaling Group Configuration

Dynamic Scaling Policy

CloudWatch Metric:

Add your answer

Relevant content