Best option to migrate from Java code in Lambda to a long running AWS Service

Question

Hello, I have 2 Lambda function2, developed in java, which implement an ETL doing 2 things: 1\ (first lambda) copying files to an S3 bucket and then 2\ (second lambda) doing some processing on the file once copied.
Each lambda is managing a different file.

The issue is that now the step 2\ is starting to hit the 15 minutes lambda limit, due to big size of the file.

Which is the easier and quicker option to rearchitect this lambda/job using other AWS services to remove 15 min execution limit? 
I'm considering for example to put these two lambdas in two container images (with minimal changes to java code) and deploy them in ECS Fargate or App Runner.

Didier Durand · Answer

Hi,

I am a great fan of AWS Batch for jobs that you need to run on demand. It's also based on container images and very easy to set up. BTW, under the hood, it leverages Fargate.

In my experience, going to ECS / Fargate directly in more appropriate for long-lasting applications: for example a permanent web application.

AWS Batch makes it very easy to submit your jobs once in a while as you need.  Also,  the setup is much simpler than ECS/Fargate.

See https://aws.amazon.com/batch/

You may want to have a look at https://www.youtube.com/watch?v=Ym9HWYFwFS8

Best,

Didier

MarcioMangar · Answer

Hello!,

Based on what you have described about the two Lambda functions, I would suggest you take a look at Fargate.
That may take you some time to migrate your Java code to a container-based and configure the Fargate for that.

If you allow me, I would also suggest, if possible to consider changing from Java to Python (NumPy + Pandas) when handling large files.
You may take good advantage of that.

Best,
Mangar

Best option to migrate from Java code in Lambda to a long running AWS Service

Add your answer

Relevant content