Amazon ECS/SQS/lambda/

Question

Good morning everyone,

I am just starting in the AWS world and I have a challenge that I need to solve with the most appropriate tools that AWS offers me.

The use case is the following: I have to process some pdf documents add some images to them and send it back.
Currently I am doing it with a microservice that receives a pdf and returns it modified. 
When I do load tests the queue receives 50 requests and in the bash task I get blocked with 9 pdf at the same time and the ECS crashes.

One solution is to increase the capacity of the ECS so that the microservice can process more documents. But I have read that SQS can help me solve this so I want to be sure I am applying the right architecture:

- I have a .net core microservice in docker that produces requests and sends them to the queue.
- I have an SQS that receives requests and arranges them in order of arrival.
- I have a lambda that listens to the SQS and when a new request arrives it fires the event to the consuming microservice (the lambda "fires" up to 10 times simultaneously and in each "firing" it lets only 1 document through, or they recommend that in each "firing" it lets 10 documents through).
- The consuming microservice receives a message from the lambda and starts processing all the SQS requests until all of them are finished.

- When finished and the SQS is emptied the lambda again is waiting for the SQS to have a new message and the cycle starts again.

Overview:
I have a microservice is publisher.
The microservice is consumer
The lambda is the trigger
The SQS is the queue

Answer

Whether you can use SQS or not will depend on whether your requirement is to process the files synchronously or you are OK processing them asynchronously.

If asynchronous processing is fine, then yes SQS can help you buffer requests so your back-end can process the requests.

You did mention Lambda and .NET. Hopefully your framework is .NET Core because Lambda cannot run .NET Framework as Lambda microVMs are essentially Linux based.

If the above conditions are met and you want to use Lambda with SQS, then please go through this document and understand how Lambda works with SQS thoroughly, because with SQS, Lambda polls batches of messages and there are some parameters such as batch size and batch window and queue visibility timeout that you have to configure correctly to get the best performance - https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

Also if you are handling large files, use a combination of SQS and S3, as mentioned here - https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-s3-messages.html

Amazon ECS/SQS/lambda/

Relevant content