How can I stop my Amazon ECS service from failing to stabilize in AWS CloudFormation?

4 minute read
2

My Amazon Elastic Container Service (Amazon ECS) service fails to stabilize in AWS CloudFormation. I get the following error: "Service arn:aws:ecs:us-east-accountID:service/ServiceName did not stabilize."

Short description

A service created in Amazon ECS fails to stabilize if it isn't in the state specified by the AWS CloudFormation template. To confirm that a service launched the desired number of tasks with the desired task definition, AWS CloudFormation makes repeated DescribeService API calls. These calls check the status of the service until the desired state is met. The calling process can take up to three hours. Then, AWS CloudFormation times out, and returns the "Service ARN did not stabilize" message. While AWS CloudFormation checks the status of the service, the stack that contains the service remains in the CREATE_IN_PROGRESS or UPDATE_IN_PROGRESS state and can't be updated.

If you can't fix the underlying issue with your Amazon ECS service tasks immediately and you don't want to wait for the DescribeService API calls to time out, then you can manually force the state of the Amazon ECS service resource in AWS CloudFormation into a CREATE_COMPLETE state. To do this, manually set the desired count of the service to zero in the Amazon ECS console to stop running tasks. AWS CloudFormation then considers the update as successful, because the number of tasks equals the desired count of zero.

Important: Manually forcing AWS CloudFormation into a CREATE_COMPLETE state isn't a best practice for production services, because all tasks are stopped, and doing this can cause a production outage.

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.

Resolution

Verify resource creation

1.    In your AWS CloudFormation template, create an AWS::ECS::Service resource. For example:

Resources:
ECSServiceA:
 Type: AWS::ECS::Service
Properties:
  DesiredCount: 1
  Cluster: awsExampleECSCluster
  LaunchType: EC2
  ServiceName: "MyNginxService2"
  TaskDefinition: NginxTask:1

2.    Open the AWS CloudFormation console, and then select your stack.

3.    Choose the Events tab, and then verify that your resource is being created.

Update the desired count of the service

You can update the desired count of the service to your original value with either the AWS CLI or Amazon ECS console.

Using the AWS CLI:

1.    To describe the service and list the service events, run the following command:

aws ecs describe-services --cluster awsExampleECSCluster --services MyNginxService2

2.    To update the desired count of the service, run the following command:

aws ecs update-service --cluster awsExampleECSCluster --service MyNginxService2 --desired-count 0

3.    Update --desired-count to your original value.

Using the Amazon ECS console:

1.    Open the Amazon ECS console.

2.    In the navigation pane, choose Clusters, and then select the cluster that contains the Amazon ECS service that you created.

3.    On the Clusters page, choose the cluster that contains the Amazon ECS service that you created.

4.    On the page for the cluster that you selected, in the Service Name column, choose your service.

5.    Choose the Events tab, and then choose Update.

6.    On the Configure service page, for Number of tasks, enter 0.

7.    Choose Next step to step through to the end of the Update Service wizard, and then choose Update Service.

The service now reaches a steady state and transitions the Amazon ECS service resource in AWS CloudFormation to CREATE_COMPLETE or UPDATE_COMPLETE.

Important: To make your AWS CloudFormation stack sync with the Amazon ECS service properties after you fix the issue with the underlying tasks, you must manually change the desired count (DesiredCount) back to the original value from your template.


Related information

Updating a service

update-service

services-stable

AWS OFFICIAL
AWS OFFICIALUpdated 3 years ago