Help us improve the AWS re:Post Knowledge Center by sharing your feedback in a brief survey. Your input can influence how we create and update our content to better support your AWS journey.
Best option to migrate from Java code in Lambda to a long running AWS Service
Hello, I have 2 Lambda function2, developed in java, which implement an ETL doing 2 things: 1\ (first lambda) copying files to an S3 bucket and then 2\ (second lambda) doing some processing on the file once copied. Each lambda is managing a different file.
The issue is that now the step 2\ is starting to hit the 15 minutes lambda limit, due to big size of the file.
Which is the easier and quicker option to rearchitect this lambda/job using other AWS services to remove 15 min execution limit? I'm considering for example to put these two lambdas in two container images (with minimal changes to java code) and deploy them in ECS Fargate or App Runner.
- 语言
- English
- 最新
- 投票最多
- 评论最多
Hi,
I am a great fan of AWS Batch for jobs that you need to run on demand. It's also based on container images and very easy to set up. BTW, under the hood, it leverages Fargate.
In my experience, going to ECS / Fargate directly in more appropriate for long-lasting applications: for example a permanent web application.
AWS Batch makes it very easy to submit your jobs once in a while as you need. Also, the setup is much simpler than ECS/Fargate.
See https://aws.amazon.com/batch/
You may want to have a look at https://www.youtube.com/watch?v=Ym9HWYFwFS8
Best,
Didier
Hello!,
Based on what you have described about the two Lambda functions, I would suggest you take a look at Fargate. That may take you some time to migrate your Java code to a container-based and configure the Fargate for that.
If you allow me, I would also suggest, if possible to consider changing from Java to Python (NumPy + Pandas) when handling large files. You may take good advantage of that.
Best, Mangar
