How to I troubleshoot a deploy that is forever in "IN_PROGRESS"?

0

I'm using ECS with Fargate, I have a service running and it is working OK. But after I update the task definition (new deploy) and the console (ECS -> Clusters -> Tasks tab) shows that my current task is INACTIVE, which is normal, but it doesn't show any new task being created, nor any stopped task, even after an hour. It is as if ECS is not trying to run my new definition.

If I use the awscli to find information about my service:

aws ecs describe-services --cluster cluster-xxxxxxx --services service-svc-xxxxxxx --region us-east-1
It has two deployments. The first is alright, it is the running deployment. The most recent one, it shows:

          "desiredCount": 1,  
          "pendingCount": 0,  
          "runningCount": 0,  
          "failedTasks": 7,  
          ...  
          "rolloutState": "IN_PROGRESS",  
          "rolloutStateReason": "ECS deployment ecs-svc/XXXXXXXXXXXXXXXXXX in progress."  

Again, there is nothing on the ECS console that points to failed tasks. It is as if the task is failing on a so premature state, its not even logging anything. On the "Task" tab, when I point "Desired task status:" to "Stopped" it shows an empty list with "No Results".

I tried looking at CloudTrail event, but there is nothing there about failed tasks. On CloudWatch, the logs for container insights (/aws/ecs/containerinsights/cluster-xxxxxxx/performance) also don't mention failing tasks.

How can I troubleshoot this situation?

diogofc
asked 2 years ago1901 views
1 Answer
0

Turns out AWS is applying quotas without specifying it anywhere on the console or documentation. And there seems to be a "trick", to overcome this, which is to create an EC2 instance on the same region you are trying to use ECS.

https://forums.aws.amazon.com/thread.jspa?threadID=340770

diogofc
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions