ECS Service Creation is Stuck in "CREATE_IN_PROGRESS" Status

0

I have created an ECS Cluster , then I created a new Service based on a Task Definition through the AWS Management Console , the Problem Is that Its status Is CREATE_IN_PROGRESS In CloudFormation Console for more that an hour and It doesn't show any error , and I don't know if this Is considered normal or not , what should I do to know If there Is an issue or if this Is normal ?

I don't know where to Investigate as I don't know what might be the cause of this Issue , I have checked CloudWatch logs and I can't see any logs , though in the task definition I am specifying a cloud watch group to log into

Mahmoud
asked 14 days ago102 views
2 Answers
2
  1. Inspect ECS Task Status: Sometimes, the service creation might be stuck because the tasks defined in the task definition cannot start. Go to the ECS console, select your cluster, and look at the "Tasks" tab to see if the tasks are stuck in a PENDING state or failing to launch.

  2. Task Definition Issues: Verify that the definition is correctly configured. Issues such as invalid container image URIs, insufficient CPU or memory allocation, or incorrect networking configuration can prevent tasks from starting.

  3. Subnet and Security Group Configuration: Ensure that the subnets and security groups specified in your service definition allow the ECS tasks to communicate with necessary AWS services and external networks as needed. Incorrect network configurations can prevent tasks from being placed.

  4. IAM Roles and Policies: Check that the IAM roles associated with the ECS tasks, services, and the ECS cluster have the necessary permissions. Insufficient permissions can block tasks from accessing AWS resources they require to function, like ECR for images or logs for CloudWatch.

  5. ECS Service Role: Ensure that the ECS service role (ecsServiceRole) has the appropriate permissions. This role needs policies that allow it to make calls to ELB, VPC, and ECR on your behalf.

  6. AWS Support: If you still cannot determine the cause after checking the above, consider opening a support case with AWS. AWS support can provide more detailed insights based on the internal state of the resources being deployed.

profile picture
EXPERT
answered 13 days ago
profile picture
EXPERT
Artem
reviewed 8 days ago
0

The issue you're experiencing with your ECS service creation being stuck in the CREATE_IN_PROGRESS status is likely due to the service not stabilizing, as mentioned in the article AWS Knowledge Center - CloudFormation ECS Service Stabilize. The article provides the following information:

A service created in Amazon ECS fails to stabilize if it isn't in the state specified by the AWS CloudFormation template. To confirm that a service launched the desired number of tasks with the desired task definition, AWS CloudFormation makes repeated DescribeService API calls. These calls check the status of the service until the desired state is met. The calling process can take up to three hours. Then, AWS CloudFormation times out, and returns the "Service ARN did not stabilize" message.

In your case, it seems that the ECS service is not reaching the desired state within the timeout period, causing the CloudFormation stack to remain in the CREATE_IN_PROGRESS status.

To investigate and potentially resolve this issue, you can follow the steps outlined in the "Resolution" section of the article:

  1. Verify resource creation: Ensure that the AWS::ECS::Service resource is correctly defined in your CloudFormation template, as shown in the example provided in the article.

  2. Update the desired count of the service: If the underlying issue with your ECS service tasks cannot be fixed immediately, you can try manually updating the desired count of the service to 0, as described in the article. This will force the CloudFormation stack to transition to the CREATE_COMPLETE or UPDATE_COMPLETE state.

    You can update the desired count using either the AWS CLI or the Amazon ECS console, as detailed in the article.

    # Using the AWS CLI
    aws ecs update-service --cluster awsExampleECSCluster --service MyNginxService2 --desired-count 0
    

    Important: Manually forcing the CloudFormation stack to a CREATE_COMPLETE state by stopping all tasks is not recommended for production services, as it can cause a temporary outage.

  3. Monitor the service and update the desired count: After updating the desired count to 0, the CloudFormation stack should transition to CREATE_COMPLETE or UPDATE_COMPLETE. You can then update the desired count back to your original value to resume normal operation.

If the issue persists or you don't see any related logs in CloudWatch, you may need to investigate further by checking the ECS service events, Tasks events and logs, and any other relevant AWS services to identify the root cause of the stabilization issue.

profile pictureAWS
answered 13 days ago
  • Thanks For Your Answer , I have tried this approach , and Indeed after setting the count to 0 , the Service stack Is Created Successfully. But after I set it back to the desired count of tasks I.e. 1 , the Task Is never created. I have another question why does cloudwatch logs doesn't log anything related so that I can Investigate further ?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions