Cannot deploy to Fargate with 4 tasks - Limit reached for concurrent tasks

0

Hello!

I have 2 ECS Fargate clusters with 1 service running in each of them. For each service, it is configured to run with 2 tasks.

I have been trying to deploy a new revision of the service to one of the clusters, and it fails with the following error:

"""
service <redacted> was unable to place a task. Reason: You've reached the limit on the number of tasks you can run concurrently. For more information, see the Troubleshooting section.
"""

Troubleshooting section links to http://docs.aws.amazon.com/AmazonECS/latest/developerguide/troubleshooting.html

I don't understand what limits did I breach here, as I only have 4 tasks running in total between the two clusters. Based on the quotas mentioned here https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-quotas.html, I am no where any of those limits.

I have tried re-deploying, but the service is stuck with the above mentioned error.

Any one has any suggestions on how to resolve this?

Thanks!

Edited by: CrystalStarburst on May 21, 2021 2:53 AM

asked 3 years ago1694 views
5 Answers
1

We encountered this too. The quota limits for our account were 50 for on-demand and spot respectively (aws service-quotas list-service-quotas --service-code fargate). However, I could only launch two tasks concurrently.

I reached out to support, and their advice was to launch an EC2 instance (with whatever configuration) into the region your tasks are in. This seemed to solve it for us. Apparently when using Fargate in a new region, the service might run into concurrency issues.

JTH
answered 2 years ago
0

Hi CrystalStarburst,

I can't offer any help unfortunately, but I just wanted to weigh in to say we have the same thing with our AWS account.

What I find interesting is that you posted this on the 21st May, and we ran into this issue on the 20th May. In our case we have a single cluster with two services running. Git branches are deployed to new services which are torn down automatically when the branch is merged, so the total number of tasks would vary from 4 to around 16. On the 20th, new deployments failed with the error "...was unable to place a task. Reason: You've reached the limit on the number of tasks you can run concurrently. For more information, see the Troubleshooting section", and since then the most we've been able to create is 2 tasks in Fargate On-Demand, and 2 tasks in Fargate Spot.

Nothing has changed with our system, and searching on the web has shown people saying that there's an internal limit applied to newer AWS accounts (some AWS doc pages refer to these "invisible" limits), so at the moment we're just assuming AWS changed something.

Were you able to spin up more than 4 tasks before 20th May, do you know?

  • Matt
answered 3 years ago
0

Hi CrystalStarburst,

Just to follow up with how we managed to fix this. We requested a quota increase for the number of Fargate On-Demand resource count. Although it was listed as something like 100, it was obviously silently limiting us to 2 (4 in total across Fargate). There was a bit of back-and-forth where we had to estimate total vCPU/memory requirements of our tasks, time to create, etc, but they eventually increased it and now all of our services are running again.

Nice to have it resolved, but a bit of a worry that AWS basically tore down our production account with no communication whatsoever.

Hope you were able to fix yours!

-Matt

answered 3 years ago
0

Just wanted to add that I too had this problem for at least the last several days. The default value for fargate is 1000, somehow my account had it set for 100, but I had actually been limited to 2. I spent a ridiculous amount of time debugging why my changes were being updated.

RTH
answered 3 years ago
0

same on my account default 1000 my account has 100 and limit to 2

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions