I have deployed a ECS Cluster through CloudFormation and I used a "AWS::ECS::ClusterCapacityProviderAssociations" resource to set up a capacity provider. After adding services to the cluster I'm unable to update the AMI due to the following error:
Resource handler returned message: "Out of retries. Last encountered error was: The specified capacity provider is in use and cannot be removed. (Service: AmazonECS; Status Code: 400; Error Code: ResourceInUseException; Request ID: bfcb7eb4-92de-4f78-9ebd-cc13a5b38f2c; Proxy: null)" (RequestToken: db9bb9e9-f733-053d-f4cb-3a1437834715, HandlerErrorCode: null)
I suspect this is because the old capacity provider is not drained of tasks before applying the new one. I tried utilizing the UpdatePolicy attribute for ASG without any luck.
I was able to reproduce the issue with an example from the docs:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/quickref-ecs.html#create-cluster-al2023
Including the template here for completness:
AWSTemplateFormatVersion: 2010-09-09
Description: EC2 ECS cluster that starts out empty, with no EC2 instances yet.
An ECS capacity provider automatically launches more EC2 instances as required
on the fly when you request ECS to launch services or standalone tasks.
Parameters:
InstanceType:
Type: String
Description: EC2 instance type
Default: "t2.medium"
AllowedValues:
- t1.micro
- t2.2xlarge
- t2.large
- t2.medium
- t2.micro
- t2.nano
- t2.small
- t2.xlarge
- t3.2xlarge
- t3.large
- t3.medium
- t3.micro
- t3.nano
- t3.small
- t3.xlarge
DesiredCapacity:
Type: Number
Default: "0"
Description: Number of EC2 instances to launch in your ECS cluster.
MaxSize:
Type: Number
Default: "100"
Description: Maximum number of EC2 instances that can be launched in your ECS cluster.
ECSAMI:
Description: The Amazon Machine Image ID used for the cluster
Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
Default: /aws/service/ecs/optimized-ami/amazon-linux-2023/recommended/image_id
VpcId:
Type: AWS::EC2::VPC::Id
Description: VPC ID where the ECS cluster is launched
Default: vpc-1234567890abcdef0
SubnetIds:
Type: List<AWS::EC2::Subnet::Id>
Description: List of subnet IDs where the EC2 instances will be launched
Default: "subnet-021345abcdef67890"
Resources:
# This is authorizes ECS to manage resources on your
# account on your behalf. This role is likely already created on your account
# ECSRole:
# Type: AWS::IAM::ServiceLinkedRole
# Properties:
# AWSServiceName: 'ecs.amazonaws.com'
# ECS Resources
ECSCluster:
Type: AWS::ECS::Cluster
Properties:
ClusterSettings:
- Name: containerInsights
Value: enabled
# Autoscaling group. This launches the actual EC2 instances that will register
# themselves as members of the cluster, and run the docker containers.
ECSAutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
DependsOn:
# This is to ensure that the ASG gets deleted first before these
# resources, when it comes to stack teardown.
- ECSCluster
- EC2Role
Properties:
VPCZoneIdentifier:
Ref: SubnetIds
LaunchTemplate:
LaunchTemplateId: !Ref ContainerInstances
Version: !GetAtt ContainerInstances.LatestVersionNumber
MinSize: 0
MaxSize:
Ref: MaxSize
DesiredCapacity:
Ref: DesiredCapacity
NewInstancesProtectedFromScaleIn: true
UpdatePolicy:
AutoScalingReplacingUpdate:
WillReplace: "true"
# The config for each instance that is added to the cluster
ContainerInstances:
Type: AWS::EC2::LaunchTemplate
Properties:
LaunchTemplateName: "asg-launch-template"
LaunchTemplateData:
ImageId:
Ref: ECSAMI
InstanceType:
Ref: InstanceType
IamInstanceProfile:
Name: !Ref EC2InstanceProfile
SecurityGroupIds:
- !Ref ContainerHostSecurityGroup
# This injected configuration file is how the EC2 instance
# knows which ECS cluster on your AWS account it should be joining
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
echo ECS_CLUSTER=${ECSCluster} >> /etc/ecs/ecs.config
yum install -y aws-cfn-bootstrap
/opt/aws/bin/cfn-init -v --stack ${AWS::StackId} --resource ContainerInstances --configsets full_install --region ${AWS::Region} &
# Disable IMDSv1, and require IMDSv2
MetadataOptions:
HttpEndpoint: enabled
HttpTokens: required
EC2InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles:
- !Ref EC2Role
# Create an ECS capacity provider to attach the ASG to the ECS cluster
# so that it autoscales as we launch more containers
CapacityProvider:
Type: AWS::ECS::CapacityProvider
Properties:
AutoScalingGroupProvider:
AutoScalingGroupArn: !Ref ECSAutoScalingGroup
ManagedScaling:
InstanceWarmupPeriod: 60
MinimumScalingStepSize: 1
MaximumScalingStepSize: 100
Status: ENABLED
# Percentage of cluster reservation to try to maintain
TargetCapacity: 100
ManagedTerminationProtection: ENABLED
# Create a cluster capacity provider assocation so that the cluster
# will use the capacity provider
CapacityProviderAssociation:
Type: AWS::ECS::ClusterCapacityProviderAssociations
Properties:
CapacityProviders:
- !Ref CapacityProvider
Cluster: !Ref ECSCluster
DefaultCapacityProviderStrategy:
- Base: 0
CapacityProvider: !Ref CapacityProvider
Weight: 1
# A security group for the EC2 hosts that will run the containers.
# This can be used to limit incoming traffic to or outgoing traffic
# from the container's host EC2 instance.
ContainerHostSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Access to the EC2 hosts that run containers
VpcId:
Ref: VpcId
# Role for the EC2 hosts. This allows the ECS agent on the EC2 hosts
# to communciate with the ECS control plane, as well as download the docker
# images from ECR to run on your host.
EC2Role:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- sts:AssumeRole
Path: /
ManagedPolicyArns:
# See reference: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/security-iam-awsmanpol.html#security-iam-awsmanpol-AmazonEC2ContainerServiceforEC2Role
- arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
# This managed policy allows us to connect to the instance using SSM
- arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
# This is a role which is used within Fargate to allow the Fargate agent
# to download images, and upload logs.
ECSTaskExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
Action:
- sts:AssumeRole
Condition:
ArnLike:
aws:SourceArn: !Sub arn:${AWS::Partition}:ecs:${AWS::Region}:${AWS::AccountId}:*
StringEquals:
aws:SourceAccount: !Sub ${AWS::AccountId}
Path: /
# This role enables all features of ECS. See reference:
# https://docs.aws.amazon.com/AmazonECS/latest/developerguide/security-iam-awsmanpol.html#security-iam-awsmanpol-AmazonECSTaskExecutionRolePolicy
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
Outputs:
ClusterName:
Description: The ECS cluster into which to launch resources
Value: ECSCluster
ECSTaskExecutionRole:
Description: The role used to start up a task
Value: ECSTaskExecutionRole
CapacityProvider:
Description: The cluster capacity provider that the service should use to
request capacity when it wants to start up a task
Value: CapacityProvider
I deployed the template like so:
aws cloudformation deploy --stack-name tmp --template-file cluster.yml --capabilities CAPABILITY_IAM --parameter-overrides MaxSize=5 VpcId=vpc-REDACTED SubnetIds=subnet-REDACTED
Then I deployed a service to the cluster that is using the default capacity provider.
And finally I made an update that changes the AMI (I picked a AMI at random):
aws cloudformation deploy --stack-name tmp --template-file cluster.yml --capabilities CAPABILITY_IAM --parameter-override ECSAMI=/aws/service/ecs/optimized-ami/amazon-linux-2023/ami-022198f01f39f2101/image_id
This fails when updating the CapacityProviderAssociation resource:
Resource handler returned message: "Out of retries. Last encountered error was: The specified capacity provider is in use and cannot be removed.
Am I missing something here?
That's the whole point of this stack