Serverless vs EKS/EC2
We are building a service which will be exposed as Rest API through API Gateway. The service will perform 3 things.
- Validate the inputs
- Upload a document to S3(rather move the document-approx 40kb in size from Staging S3 location to Target S3)
- Insert the metadata to Dynamo DB
We expect this service to be called approx 2 lakh times within 8-10 hours on daily basis through AWS Batch. Afterwards, it may not be used through out the day. We were planning to have a lambda function for each of the above steps which will be called from API Gateway. If required, we may use Step functions to orchestrate above 3 steps.
Would Lambda functions be the appropriate way to process these steps in the service or should we use EC2/EKS and ASG to scale in and Scale out.
I assume the service will respond back within second but bearing Lambda concurrency limits in mind and any other limits we may face, let me know the best way to process the batch.
Regards, Dhaval Mehta
Yes, using Lambda here, see documentation - https://docs.aws.amazon.com/lambda/latest/dg/services-s3-batch.html
There is also a blog post on batch processing - https://aws.amazon.com/blogs/compute/creating-aws-serverless-batch-processing-architectures/
And some best practices - https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html
If you build your Lambda function with containers, then you can consider running the containers in EKS in future should your use case change.
You can't really just use the same container image and run it on a traditional container solution (ECS/EKS/Docker/etc). Lambda has a specific event driven programming model that do not exist on the other platforms. The container image is just a packaging mechanism (what runs inside the image is different between Lambda and the other platforms)
Yes; the invocation mechanisms will be different. The core remains the same though.
To me this is a perfect use case for Lambda. +1
what are some ways/alternative to expose sagemaker endpoints as a HTTP /REST endpoints?asked 3 months ago
Serverless vs EKS/EC2Accepted Answerasked 5 months ago
Uploading greater than 6 MB file to S3 through API Gateway results in Request too long error, Is that expected ?asked 2 months ago
AWS Api Gateway: how to expose new services automatically?asked 3 months ago
How to use postgres rest api in AWSasked 5 days ago
Api Gateway Payload size limit increaseasked 5 years ago
Is it possible to send json data in body of GET method defined in API Gateway REST API ?Accepted Answerasked 2 months ago
For S3 Intelligent Tiering which exact API operations are required for an object to be marked as `Accessed`?asked 2 months ago
Best practices for securing service to service REST communication in a microservices architectureAccepted Answerasked 2 years ago
Looking for a best-practice for building serverless application with amplify and api gateway and lambdaAccepted AnswerMODERATORasked 3 years ago