By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Limit the number of task running in AWS cluster at a time

0

I have a structure in which an event bridge triggers an ECS task when a message arrives in SQS. I want to limit the number of tasks running at the same time.

Example: If there are 100 messages coming in the queue at the same time, all should not get triggered at the same time. So i will need a mechanism such that only 10 tasks should run at a time, if one of the task is completed then next task can be taken from queue.

Topics
ServerlessApplication IntegrationContainers
Tags
Amazon EventBridgeAmazon Simple Queue ServiceAmazon Elastic Container Service
Language
English
HingeAjit
asked 6 months ago264 views
1 Answer
0

Could you please provide more context on how you are using EventBridge to launch the ECS task, as well as information on the ECS infrastructure (EC2 or Fargate launch type)?

Typically, limiting the number of tasks running at a time will involve running the task within a service with auto-scaling settings. For example, the service could have a minimum of 0 tasks and a maximum of 10, and use a scaling policy based on the size of the SQS queue. A similar approach can be seen here: https://serverlessland.com/patterns/eventbridge-sqs-ecs-cdk?ref=search Another example discussing ECS auto-scaling here: https://aws.amazon.com/blogs/containers/amazon-elastic-container-service-ecs-auto-scaling-using-custom-metrics/

AWS
jthwaits
answered 6 months ago
  • HingeAjit
    6 months ago

    I have an event bridge which triggers an ECS task when a message comes in SQS. So everytime a message comes in SQS it will trigger an ECS task. So i wanted to limit the number of tasks running concurrently so to manage the resources of ECS Fargate service.

  • jthwaits
    6 months ago

    There is currently no native way to manage the max concurrency of ECS tasks being launched in a cluster from an EventBridge rule. It should be possible to re-architect the solution to operate in a manner similar to what is described in the serverlessland linked pattern: Create an ECS service with autoscaling (minimum 0, maximum 10) and set scaling policies based on the length of the SQS queue. In this way, as items are added to the queue, a 1:1 relationship of running tasks to queue messages will exist up to a maximum of 10 tasks.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content