AWS Service that can act like Lambda without a time limit?

Question

My web app processes photos on a large scale. A user might upload 20,000 images at once to be processed. I upload them to S3 and then process them.

Like Lambda allows, I need to be able to trigger the processing of the photos via API Gateway, have multiple instances running at once that are isolated from each other, and when the instance is done running it stops with the ability to scale to 0 when there is no demand (like at night time) or scale to virtually unlimited if there are 10,000 that upload photos at the same time.

Lambda would be perfect for this but with the 15 minute limit it will not work as some jobs could take up to 2 hours to finish.

Is there another AWS service that would allow me to do what I need but not have the time limit? I looked into Fargate and Batch but they do not seem to really work for the requirements.

Answer

I suggest using your preferred trigger s3, eventbridge or sqs/sns to start a step function which can then start a [fargate task](https://docs.aws.amazon.com/step-functions/latest/dg/connect-ecs.html) (Which does not have 15min time limit). You can also add additional logic and error handling to the step function

Answer

Hi -

What you are looking for is an equivalent of GCP's[ Cloud Run](https://cloud.google.com/run).

While I'm curious what others have to suggest, there is no **exact** Cloud Run equivalent in AWS.

There are couple of things -

1. You can have a small Fargate task running, and scale up when requests comes in; or simply use App Runner. This approach while not giving you absolutely 0 cost when there is no request, the overhead is very small; 
2. You can also use AWS Batch, Lambda, SQS, Step Function etc. to process the upload in an asynchronous manner, this is the closest to what you want to achieve. AWS Batch is made for the use case you described, not sure why you believe it would not work for you;
3. You also have the option to write your own logic, for example, use Lambda function to trigger a set of automation that creates let's say a new ECS task, and at the end of processing, trigger another Lambda function within the task that deletes the task.

AWS Service that can act like Lambda without a time limit?

Relevant content