I have a Spring Boot Java application that consumes messages from an Amazon SQS FIFO queue using the @SQSListener annotation. My consumer is multi-threaded, and I have maxConcurrentMessages set to 5, allowing multiple threads to process messages simultaneously.
I understand that SQS FIFO queues maintain the order of messages within the same MessageGroupId and deliver them in the correct sequence. However, I’m concerned that the multi-threaded nature of my consumer might break the FIFO processing guarantees, even though the messages are received in order.
For example, if I have 5 messages (message1 to message5) from the same group (groupA) in the queue, they are delivered in order and can all be in-flight at the same time in the consumer. Since my consumer uses multiple threads, these messages could be split across 5 different threads. My worry is that this setup could cause the messages to be processed out of order, breaking the FIFO guarantee of the queue.
My Questions Are:
- Can the multi-threaded consumer potentially break FIFO processing within the same MessageGroupId, even if messages are delivered in order and are in-flight simultaneously?
- What is the best way to ensure that messages within the same group are processed in the correct order, even with a multi-threaded consumer setup?
- Are there any best practices, synchronization techniques, or specific configurations that can help maintain strict ordering of messages in a multi-threaded environment?
I appreciate any insights or advice on how to manage this scenario effectively!
Thanks for the response. That makes sense. I think the issue could be that some parts of consumer logic are async (web flux), and it is using auto acknowledgement/deletion on message success. So thread1 could pick up message1 and then go in an async block and gets acknowledged. Because message1 has been acknowledged/deleted, thread2 goes on to process message2 while message1 is still running in its async block. It might need to acknowledge messages manually or introduce some synchronicity