Running Update Service shows a pending task, but never shuts down the existing task so a new deployment runs

The behavior you're experiencing is likely due to a combination of factors in your ECS configuration and deployment strategy. Here are some potential reasons and solutions:

1. Resource Constraints: If your EC2 instance doesn't have enough resources (CPU, memory, or ports) to run both the existing and new task simultaneously, the new task will remain in a pending state. ECS tries to place the new task, but can't due to insufficient resources.

2. Port Conflicts: If your container is using host networking mode and binding to specific ports, the new task might not be able to start because the required port is already in use by the existing task.

3. Deployment Configuration: While your min healthy percent (100%) and max percent (200%) settings should theoretically allow for two tasks to run simultaneously during deployment, other factors might be preventing this from happening effectively.

4. Task Placement: ECS might be unable to find a suitable placement for the new task due to task placement strategies or constraints.

To address these issues and improve your deployment process:

1. Ensure your EC2 instance has enough resources to run both tasks simultaneously during deployment.

2. If using host networking, consider using dynamic port mapping or the awsvpc network mode to avoid port conflicts.

3. Implement a rolling update strategy in your ECS service. This gradually replaces old tasks with new ones, allowing for smoother deployments.

4. Review your task placement strategies and constraints to ensure they're not preventing the new task from starting.

5. Consider implementing service auto scaling to automatically adjust the desired count of tasks based on metrics, which can help manage capacity during deployments.

6. Enable managed instance draining in your capacity provider settings. This allows ECS to gracefully remove tasks from an instance that's being scaled in.

7. If using an Application Load Balancer, configure a deregistration delay for your target group. This gives time for in-flight requests to complete before an instance is removed from service.

If these steps don't resolve the issue, you may need to investigate further by checking ECS service events, reviewing CloudWatch logs, and ensuring that your IAM roles and policies are correctly configured for ECS tasks and services.
Sources
ECS Task Deployment Doesn't Reserve More Capacity | AWS re:Post
Amazon ECS task stuck in a lifecycle state | AWS re:Post

Hi,

In ideal scenario, when you update a service by forcing new deployment, it should spin up a new task and terminate the old existing task. In this case since the new task is terminating while keeping the old task, we need to review your cluster service configuration and task specific logs to determine why the new task is getting stopped.

I would request you to create a support case from the AWS support centre dashboard with ECS team, including the resource details like cluster name, service name and task details, as this will allow us to provide you resource specific guidance and dive deep into the same.

Thank you!