Multiple Fargate tasks being created when only requesting 1

0

I've created a Lambda (NodeJS) which triggers on a file being uploaded to an S3 bucket. The Lambda uses the ecs.runTask method in the AWS SDK (v2) to then create a Fargate task using a TaskDef which includes two containers to process the file being uploaded. The task is created in an ECS cluster with no services defined.

My issue is that although I'm requesting only a single task to be created I'm always seeing two tasks being created. They both run and produce logs indicating they have processed the file which was uploaded (but I only want to process the file once).

The CloudWatch logs for the Lambda appear normal with only details for one of the tasks being returned (with the taskArn of one of the created tasks).

I've tried removing one of the containers from the TaskDef and get the same behaviour (ie. two tasks, each containing 1 container). I've tried updating the count of tasks to create to 3 and I get 6 tasks started (the Lambda logs show 3 taskARNs). (The duplicate tasks are always created within a second or two of when the first one is created.) If I create the task from the management console using the same setup, it only creates 1 task as expected.

Any suggestions on what the issue might be or how to debug it further would be appreciated.

asked 2 years ago618 views
1 Answer
0
Accepted Answer

Hard to answer without digging a lot - Are you sure the lambda is only executing once...its also possible some part of your code is making two calls (but only logging once)

Using lambda to start fargate tasks to process files in s3 is not a great pattern as error handling etc is difficult and I would suggest using step functions which can be triggered via event bridge and can synchronously start fargate tasks and have all the required logic/error handling

AWS
EXPERT
Peter_G
answered 2 years ago
  • Switched to using event bridge and step functions and that worked much better. Thanks.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions