Skip to content

Amazon SQS FIFO Queue with Multi-Threaded Consumer - Ensuring Order Per MessageGroupId

0

I have a Spring Boot Java application that consumes messages from an Amazon SQS FIFO queue using the @SQSListener annotation. My consumer is multi-threaded, and I have maxConcurrentMessages set to 5, allowing multiple threads to process messages simultaneously.

I understand that SQS FIFO queues maintain the order of messages within the same MessageGroupId and deliver them in the correct sequence. However, I’m concerned that the multi-threaded nature of my consumer might break the FIFO processing guarantees, even though the messages are received in order.

For example, if I have 5 messages (message1 to message5) from the same group (groupA) in the queue, they are delivered in order and can all be in-flight at the same time in the consumer. Since my consumer uses multiple threads, these messages could be split across 5 different threads. My worry is that this setup could cause the messages to be processed out of order, breaking the FIFO guarantee of the queue.

My Questions Are:

  1. Can the multi-threaded consumer potentially break FIFO processing within the same MessageGroupId, even if messages are delivered in order and are in-flight simultaneously?
  2. What is the best way to ensure that messages within the same group are processed in the correct order, even with a multi-threaded consumer setup?
  3. Are there any best practices, synchronization techniques, or specific configurations that can help maintain strict ordering of messages in a multi-threaded environment?

I appreciate any insights or advice on how to manage this scenario effectively!

asked 2 years ago979 views

1 Answer
2

If you process ordered messages in parallel, you will most probably loose the order, just as you indicated. The only way to ensure that the messages are processed in order is to handle each message group in its own thread. Do not split messages from the same message group to multiple threads.

You will need to implement this in the consumer itself. One way to overcome this is to use a single thread in the consumer, and run multiple consumers instead. If you do that, you will never get messages from the same message group in multiple consumers.

AWS
EXPERT

answered 2 years ago

EXPERT

reviewed 2 years ago

EXPERT

reviewed 2 years ago

  • Thanks for the response. That makes sense. I think the issue could be that some parts of consumer logic are async (web flux), and it is using auto acknowledgement/deletion on message success. So thread1 could pick up message1 and then go in an async block and gets acknowledged. Because message1 has been acknowledged/deleted, thread2 goes on to process message2 while message1 is still running in its async block. It might need to acknowledge messages manually or introduce some synchronicity

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.