At what point is an AWS SQS queue overkill as a Lambda function's source?

Question

I am using Lambda to run a handful of functions that take anywhere from 15 - 120 seconds to complete. I don't anticipate large volumes being processed at once for the foreseeable future, let's say less than 100 concurrent Lambda functions.

I am going through the process of setting up SQS queues where my Lambda functions will pull from. Is this overkill? Is there any reason to not just invoke my Lambda functions directly (via SDK) instead of sending into SQS, to then be pulled and processed by Lambda? I can see SQS being a helpful Lambda source when there are 1000s of events being processed concurrently, but that is not my use-case.

Brettski-AWS · Answer

The nice thing about using SQS as an event source is that the event source is decoupled from the Lambda function. If there was some sort of issue on the Lambda side then the work items remain in the queue to be processed once things "return to normal" If you're invoking Lambda directly then that retry logic has to be baked in on your side.

What sort of tings could go wrong? A bug in your Lambda function. Some problem in the Lambda service. Some change that leads to the processing time taking far longer. Any of those things and more.

Plus, if you do get to the point where there are more events coming in you're already set up to handle it.

So not overkill, no.

Rollerblade7 · Answer

You could always use eventbridge if you want decoupling - create a rule the will trigger your lambda from eventbridge and then send events to eventbridge instead of to SQS. The advantage is that in the future if you do need an SQS queue you can change your target to point to the queue instead of the lambda..

I would add a dead letter queue to your lambda if you don't already.

At what point is an AWS SQS queue overkill as a Lambda function's source?

Relevant content