Same Message Group Tasks executing in parallel in Celery with Amazon SQS FIFO

Question

I have a setup where I'm using Celery as the task queue with Amazon SQS FIFO. My goal is to ensure sequential processing of tasks within the same message group ID, while allowing tasks with different message group IDs to be processed in parallel. However, despite following the recommended configurations and understanding the behavior of SQS message groups, I'm experiencing parallel processing of tasks within the same message group by multiple Celery worker processes. How can I ensure that tasks with the same message group ID are processed sequentially by a single worker process, while maintaining parallel processing for tasks with different message group IDs?

Some extra details (for reference) : For celery I haven't used --concurrency setting, so it is by default spawning 4 pool processes (no. of cores). I am passing message group id using following syntax :
` message_properties = {
        "MessageGroupId": f"{supplier_id}"
    }
    celery_task.s(
        param1, **message_properties
    ).apply_async(**message_properties)

`
and I have made sure the queue is fifo and it ends with .fifo . additional settings - { 'polling_interval': 60, 'wait_time_seconds': 10, 'visibility_timeout': 600 }
[image SS](https://i.stack.imgur.com/C0yMs.png)

Answer

Not very familiar with Celery, but my guess is that it reads messages from SQS in batches and then it distributes the batches to the different works, without honoring the message group ID. If there is a way, configure the batch size to 1. This will solve the issue. Otherwise, you will need to set your workers to 1, and then you will be only able to process a single message at a time.

Same Message Group Tasks executing in parallel in Celery with Amazon SQS FIFO

Relevant content