Amazon ECS/SQS/lambda/

0

Good morning everyone,

I am just starting in the AWS world and I have a challenge that I need to solve with the most appropriate tools that AWS offers me.

The use case is the following: I have to process some pdf documents add some images to them and send it back. Currently I am doing it with a microservice that receives a pdf and returns it modified. When I do load tests the queue receives 50 requests and in the bash task I get blocked with 9 pdf at the same time and the ECS crashes.

One solution is to increase the capacity of the ECS so that the microservice can process more documents. But I have read that SQS can help me solve this so I want to be sure I am applying the right architecture:

  • I have a .net core microservice in docker that produces requests and sends them to the queue.

  • I have an SQS that receives requests and arranges them in order of arrival.

  • I have a lambda that listens to the SQS and when a new request arrives it fires the event to the consuming microservice (the lambda "fires" up to 10 times simultaneously and in each "firing" it lets only 1 document through, or they recommend that in each "firing" it lets 10 documents through).

  • The consuming microservice receives a message from the lambda and starts processing all the SQS requests until all of them are finished.

  • When finished and the SQS is emptied the lambda again is waiting for the SQS to have a new message and the cycle starts again.

Overview: I have a microservice is publisher. The microservice is consumer The lambda is the trigger The SQS is the queue

1 個回答
1

Whether you can use SQS or not will depend on whether your requirement is to process the files synchronously or you are OK processing them asynchronously.

If asynchronous processing is fine, then yes SQS can help you buffer requests so your back-end can process the requests.

You did mention Lambda and .NET. Hopefully your framework is .NET Core because Lambda cannot run .NET Framework as Lambda microVMs are essentially Linux based.

If the above conditions are met and you want to use Lambda with SQS, then please go through this document and understand how Lambda works with SQS thoroughly, because with SQS, Lambda polls batches of messages and there are some parameters such as batch size and batch window and queue visibility timeout that you have to configure correctly to get the best performance - https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

Also if you are handling large files, use a combination of SQS and S3, as mentioned here - https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-s3-messages.html

profile pictureAWS
專家
已回答 2 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南