By using AWS re:Post, you agree to the Terms of Use
/Multiple SQS messages of the same group in one batch/

Multiple SQS messages of the same group in one batch

0

Hello,

I am getting multiple messages from a same SQS FIFO group in a single receive request with MaxNumberOfMessages set to > 2. Is this correct behavior? I don't understand, the API contract says:

When receiving messages from a FIFO queue with multiple message group IDs, Amazon SQS first attempts to return as many messages with the same message group ID as possible. This allows other consumers to process messages with a different message group ID. When you receive a message with a message group ID, no more messages for the same message group ID are returned unless you delete the message or it becomes visible.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html

My code dispatches the messages directly to the workers without any group checking logic. It looks like I either need to implement such feature, or simply make all SQS receive requests with maxMessages set to 1 which sounds like a waste of resources. I am confused, because on the very same documentation page that says above, there is this box:

It is possible to receive up to 10 messages in a single call using the MaxNumberOfMessages request parameter of the ReceiveMessage action. These messages retain their FIFO order and can have the same message group ID.

That breaks the above contract which literally says: "unless you delete the message or it becomes visible". In my testing, I see those messages immediately returned in a single receive call, yet they are set to expire in 30 seconds. I don't understand, is there some kind of log that could tell me why a message was delivered that fast? I am probably missing something in here.

Thanks for help

1 Answers
1
Accepted Answer

This is exactly how it should work. SQS FIFO maintains the order of all messages with the same group ID. To make sure that the order is preserved, if one consumer got a message with a specific group ID, no other consumer will be able to get messages with the ID until the message is deleted or becomes visible again. Otherwise, the messages can be processed out of order.

Nothing prevents a single consumer to get more than one message for the same group ID as the consumer knows the order of the messages (the order of the messages in the array) and it can process them in order.

If your code sends the messages to be processed in an asynchronous manner, you will break the order. You should process the messages one at a time. In order to process the messages quicker you should have multiple consumers, each will get messages from different groups IDs and this is how you will maintain the order and process the messages quicker.

EXPERT
answered 19 days ago
  • Thanks, makes sense. So even if a consumer receives 10 messages from a single group and only confirms (deletes) the first one and dies, SQS will re-deliver the remaining 9 after the timeout regardless they were already delivered in a batch. I guess I will process them synchronously then.

  • Correct. Even if you receive a batch of messages, only the ones that are deleted will not be visible again. The other message will become available after the visibility timeout.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions