SNS to Lambda vs SNS to SQS to Lambda

Question

I'm trying to understand whether I need SQS in my workflow if someone can help explain. In my app, when an action is taken, it submits info to SNS topic which invokes Lambda to do some processing. This is working great as it is.

When I do research online, it seems that people are using SQS in this stack as well where SNS would put info on SQS and then SQS would then invoke Lambda.

I guess what I'm trying to understand is the need for SQS in this. What value does that add? In other words, what am I losing by invoking my Lambda directly from SNS?

Accepted Answer

Here are a few things I can think of:

1. Batching: SQS uses batching to aggregate multiple events before invoking a Lambda function invocation. This can speed up processing by reusing hot code, reducing the number of Lambda function invocations (thus amortizing cold start overhead), and reducing the chance of encountering Lambda function execution concurrency limits. However, there is a possible latency tradeoff: if the event frequency is low, SQS may wait up to 20 seconds before flushing pending messages to Lambda.
2. Observability: With SQS you can easily observe the length of the backlog of events that have not been processed by your Lambda function or that have failed processing. This can be useful for determining whether something is going wrong.

Answer

### You will get the following benefits by using SQS : -

1. helps in the retry mechanism if the lambda process fails. 
2. routes the messages to Dead Letter Queue after the retry limit is reached, which enables you to process them at a later time within the next 14 days(maximum). i.e persistent
3. If you want to maintain the order of the messages then you can go with SQS FIFO.
4. Multiple subscribers can poll from the same SQS, but a message can be processed only by 1 subscriber, unlike SNS. In the SNS scenario, as it is a push mechanism, all the subscribers can consume the message and trigger their own processes.
5.  SQS primarily used for decoupling sending and receiving components.

### In case of SNS to Lambda:-

1. It is good for time-critical processes

### SNS -> SQS -> Lambda is a better approach if you need a Fan-Out De-coupled solution.

https://aws.amazon.com/blogs/aws/queues-and-notifications-now-best-friends/

SNS to Lambda vs SNS to SQS to Lambda

You will get the following benefits by using SQS : -

In case of SNS to Lambda:-

SNS -> SQS -> Lambda is a better approach if you need a Fan-Out De-coupled solution.

관련 콘텐츠