How to process on message at a time from SQS?
I have a queue of tasks that can only be performed one at a time, as they would be competing with each other for resources.
Think data pipeline, which needs to do a batch kind of task to process large amounts of data.
I am thinking of queueing up these tasks in SQS and then processing them one by one.
What's the best native way of going about it, without having to reinvent too much?
I know I can use a Lambda to read one message, and then start the process, and check the status/completion, before reading another message. But wondering if there's a better way.
Currently, our service uses a Step Function that triggers a CodeBuild (with a concurrency of 1).
So ideally, I'd like to trigger one step function on every incoming SQS message. But have other executions waiting until an active one completes.
- Newest
- Most votes
- Most comments
Just use a an SQS FIFO queue and assign the GroupID to all messages. Process the queue with a Lambda function. Because you use the same group ID, there will never be more than one consumer.
One option would be to put a lambda between the step function and queue and use it to dequeue the message and pass it to step functions for further processing. That way you can control the concurrency in the lambda.
Another option would be Within the Step Functions you can implement a logic to check how many instances of it are running. You can do this by using ListExecutions for StepFunctions with a filter for RUNNING. If there are 2 instances ( current and another one, then you can put the message back into the queue) and stop the process, else you can continue. You can refer to Serverlesspresso Workshop at https://catalog.us-east-1.prod.workshops.aws/workshops/28e7066a-b0bb-42ad-a0e9-8e8eeeb51133/en-US/1-workflow/4-capacity for more details on the Check Capacity step.
Relevant content
- asked 7 months ago
AWS OFFICIALUpdated 3 years ago- AWS OFFICIALUpdated 7 months ago
- AWS OFFICIALUpdated 6 months ago
- AWS OFFICIALUpdated 8 months ago
But the SQS source can spawn many Lambda functions for each message.
Also, we cannot run the actual process in a Lambda, as it's a long-running "build" process which happens inside CodeBuild.
So at best, we can trigger a Step Function, or CodeBuild directly, as a result of an incoming message. But if the next message is ingested, and processed, it will then have to be put back in the queue, if the process is already running, which will lead to wasteful reading and Lambda executions.
If your task runs in Lambda, what I suggested works. However, if your Lambda invokes a different process. you will need some other mechanism like Step Functions with concurrency controls that you implement using DynamoDB for example. Not that you can invoke a state machine without using Lambda by using EventBridge Pipes,