How do I troubleshoot an Amazon ECS task that’s stuck in a provisioning loop?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

To troubleshoot this issue, view your Amazon ECS service events for the affected service to check for recent deployment activities and failures. Use the event patterns and error messages to determine whether the issue is related to capacity, networking, or configuration issues. For more details about task launch failures, check the stopped task error messages for TaskFailedToStart errors.

Check your capacity provider configuration

To check whether the capacity provider's Amazon EC2 Auto Scaling group reaches its maximum capacity during task provisioning, run the following describe-auto-scaling-groups AWS CLI command:

aws autoscaling describe-auto-scaling-groups \
    --auto-scaling-group-names your-asg-name \
    --query 'AutoScalingGroups[].{DesiredCapacity:DesiredCapacity,MaxSize:MaxSize,RunningInstances:Instances[?LifecycleState==InService].InstanceId|length}'

Note: Replace your-asg-name with your EC2 Auto Scaling group name.

To view all container instances and their status, run the following list-container-instances command:

aws ecs list-container-instances \
    --cluster your-cluster-name \
    --status ACTIVE \
    --query 'containerInstanceArns[]'

To increase the capacity of your Auto Scaling group, run the following update-auto-scaling-group command:

aws autoscaling update-auto-scaling-group \ 
--auto-scaling-group-name your-asg-name \ 
--max-size new-max-size

Note: Replace your-asg-name with your EC2 Auto Scaling group name, and replace new-max-size with the updated EC2 Auto Scaling group size.

Also, make sure that the Amazon ECS agent is running and healthy on the container instances.

To further troubleshoot capacity provider errors, see How do I troubleshoot issues when I try to set up a new capacity provider or update an existing capacity provider for my Amazon ECS cluster?

Check your service configuration and task scaling

Make sure that the service's desired count matches the available infrastructure capacity in CPU and memory on the container instances. To view the service's desired count and running count, run the following describe-services command:

aws ecs describe-services \ 
--cluster your-cluster-name \ 
--services your-service-name \ 
--query 'services[].{desiredCount:desiredCount,runningCount:runningCount,pendingCount:pendingCount}'

Note: Replace your-cluster-name with your cluster name and your-service-name with your service name.

To update the service's desired count, run the following update-service command:

aws ecs update-service \
--cluster your-cluster-name \
--service your-service-name \
--desired-count new-count
--force-new-deployment

Note: Replace your-cluster-name with your cluster name, your-service-name with your service name, and new-count with your desired task count. The --force-new-deployment option forces the service update.

Check your service quotas

If you exceed your Amazon ECS service quotas, then you might encounter task issues. To troubleshoot service quota issues, see How do I resolve Amazon ECS service quota issues?

Check your networking configuration

Check your task security group rules and make sure that the subnet has available IP addresses. For private subnets without NAT Gateway, set up the required VPC endpoints for your configuration. The VPC endpoint security groups must allow inbound traffic from your task subnet CIDR.

Your security groups must also allow the following traffic:

• Outbound HTTPS (port 443) traffic to internet or NAT gateways to pull container images.
• Outbound traffic to the VPC endpoints that you use.
• Inbound traffic from the Application Load Balancer security group for configurations that use an Application Load Balancer.
• Inbound traffic between containers that need to communicate.

To check whether the Amazon ECS service subnet has the required IP address, run the following describe-subnets command:

aws ec2 describe-subnets \
    --subnet-ids subnet-abcde \
    --query 'Subnets[].{SubnetId:SubnetId,AvailableIPs:AvailableIpAddressCount,TotalIPs:CidrBlock}'

Note: Replace subnet-abcde with your subnet ID.

Make sure that you adhere to your task's network mode requirements:

• For awsvpc mode, make sure that each task has a dedicated elastic network interface.
  Note: There's a maximum quota for network interfaces for each instance type. Make sure that your container instances have available network interface attachment points for new tasks.
• For bridge mode, make sure that the docker0 bridge has enough IP addresses in its pool.
• For host mode, there aren't specific network resource constraints. However, port conflicts can occur when multiple tasks use the same host port.

Related information

Amazon ECS troubleshooting

Amazon ECS service quotas

Automatically scale your Amazon ECS service

Architect your solution for Amazon ECS