ECS cluster doesn't startup another EC2 instance when deploying

Question

I have a task I want to run, but deployment of that task when I try to update a running task never deploys.  I have a cluster that is running a single EC2 node.  When I perform an update service and force a new deployment that deployment sits idle waiting for that one instance to free up.  So in order to get it to deploy I have stop the existing task, wait for it to drain, deprovision, shutdown the tasks, etc.  Eventually, the new task runs, but only after the existing task stops.  It takes quite a long time to go through that cycle.  It's adding too much down time to be usable in production.

What I want is when a new deployment starts, the auto scaling group will start up a new EC2 instance, ECS will deploy the new task to that instance, then stop the old task, deprovision and shutdown the old EC2 instance.  I have my deployment configuration is the following:
## Deployment Configuration
Deployment controller type: ECS
Deployment strategy: Rolling update
Deployment circuit breaker: Turned on
Min and max running tasks: 100% min and 200% max

## Auto Scaling Group Configuration
Desired capacity: 1
Scaling limits (Min - Max): 1 - 2
Desired capacity type: Units (number of instances)

## Dynamic Scaling Policy
Policy type: Target tracking scaling
Enabled or disabled: Enabled
Execute policy when: As required to maintain Custom CloudWatch metric at 100
Take the action: Add or remove capacity units as required
Instances need: 300 seconds to warm up before including in metric
Scale in: Enabled

## CloudWatch Metric:

This is the metric used for autoscaling on the autoscaling group:

{
       "CustomizedMetricSpecification": {
        "MetricName": "CapacityProviderReservation",
        "Namespace": "AWS/ECS/ManagedScaling",
        "Dimensions": [
          {
            "Name": "CapacityProviderName",
            "Value": "Infra-ECS-Cluster-productioncluster-7854937b-AsgCapacityProvider-9ghdkeune74"
          },
          {
            "Name": "ClusterName",
            "Value": "production_cluster"
          }
        ],
        "Statistic": "Average"
      }
    }

Why isn't the auto scaling group resizing the instances to handle deployments?

re:Post Agent · Answer

The issue you're experiencing is related to how ECS manages task placement and capacity during deployments. Even though you have configured your service for a rolling update with 100% minimum and 200% maximum running tasks, the auto scaling group isn't automatically scaling up during deployments because of several potential factors:

1. **Resource Constraints**: When you force a new deployment, ECS tries to place the new task, but if your EC2 instance doesn't have enough resources (CPU, memory, or ports) to run both the existing and new task simultaneously, the new task will remain in a pending state.

2. **Port Conflicts**: If your container is using host networking mode and binding to specific ports, the new task might not be able to start because the required port is already in use by the existing task.

3. **Auto Scaling Trigger**: Your auto scaling group is configured to scale based on a Custom CloudWatch metric at 100, but this metric might not be triggered during a deployment if it's not directly related to the pending task state.

To address this issue and achieve the behavior you want (spinning up a new instance for deployments), consider these solutions:

1. **Adjust Capacity Provider Strategy**: Ensure you have properly configured your capacity provider strategy to manage the scaling of your EC2 instances based on task demand. Please see [How do I troubleshoot scaling issues with my Amazon ECS capacity provider? ](https://repost.aws/knowledge-center/ecs-capacity-provider-scaling)

2. **Enable Managed Instance Draining**: This allows ECS to gracefully remove tasks from an instance that's being scaled in, ensuring that new tasks can be placed on other instances.

3. **Use awsvpc Network Mode**: If possible, consider using the awsvpc network mode instead of host networking to avoid port conflicts during deployments.

4. **Review Task Placement Strategy**: The default strategy is to spread tasks across Availability Zones, but you might want to consider using a different strategy to optimize resource utilization.

5. **Implement Service Auto Scaling**: This can automatically adjust the desired count of tasks based on CloudWatch metrics, which can help ensure you have the right number of tasks running to handle your workload.

By implementing these changes, you should be able to create a more efficient deployment process that automatically scales out when needed for deployments and scales back in when the extra capacity is no longer required.  
**Sources**  
[Running Update Service shows a pending task, but never shuts down the existing task so a new deployment runs | AWS re:Post](https://repost.aws/questions/QUsDLK7QhMRFK0L_2TeSBF_Q/running-update-service-shows-a-pending-task-but-never-shuts-down-the-existing-task-so-a-new-deployment-runs)  
[ECS Task Deployment Doesn't Reserve More Capacity | AWS re:Post](https://repost.aws/questions/QUWg4dOlvpQkK_lAwMR5Qo8A/ecs-task-deployment-doesn-t-reserve-more-capacity)  
[ECS (EC2 launch type) new deployment container fails to start with no clear error message | AWS re:Post](https://repost.aws/questions/QU7vnlGH1FTcihhXMEZuseEA/ecs-ec2-launch-type-new-deployment-container-fails-to-start-with-no-clear-error-message)

ECS cluster doesn't startup another EC2 instance when deploying

Deployment Configuration

Auto Scaling Group Configuration

Dynamic Scaling Policy

CloudWatch Metric:

Add your answer

Relevant content