SQS Step Function Poller without Lamda

Question

Typical Pattern to execute Step function for an SQS Event is to create Lambda with SQS Event Source Mapping and run an express step function synchronously within Lambda. If step function fails, error can be thrown out of lambda, which keeps the message in SQS and will be retried by poller, after visibility timeout.

This process has one key disadvantage. During the step function process, Lambda also stays active and hence we will be paying for execution of both. 
Alternatives:
 If we execute step function asynchronously, 
* We can't use SQS + Lambda Retry, since we mark Lambda process as success and message is deleted from SQS.
* We also can't use FIFO SQS, since message will be removed before the event is fully processed by step function and next one is queue with group id might be picked up right away.

Question:
* Are there any alternate designs without writing our own SQS poller?
* If we need to write a poller, how can we effortlessly scale , like AWS Event Source Poller does?

Answer

I don't think this is going to be easy either way: Either you write some custom code for the poller/waiter/asynchronous process (so there's complexity there as well as cost, maintenance, etc.) or you continue to do what you're doing (having the Lambda function wait - which is cost but far less complexity).

Yes, there are ways to do this: For example, you could have the Lambda function store the status of the SQS message in a database (DynamoDB for example) so that you can track the in-progress items; but now you're recreating the queue logic in a separate place.

You hint at something but don't directly say it: It sounds like you (a) need to process the messages in order and (b) only want to process a certain number at a time. Why is that? Is there a threshold load on a back-end system that can't be exceeded?

It might be possible to provide another solution if you could give more context to what you're trying to achieve.

SQS Step Function Poller without Lamda

관련 콘텐츠