MessageDeduplicationId became the MessageId after being moved to the DLQ

0

Let's say the original message had:

  • MessageId: A
  • MessageDeduplicationId: B
    After being moved to the DLQ, MessageDeduplicationId became A. The main and DLQ is a FIFO queue.
    Here's the message from DLQ:
    {
    MessageId: 'a19e29fe-92a1-47fa-895f-7839b7e05868',
    ReceiptHandle: 'AQEBmbrbeMDcyADV/jd/xnCNihL1rB57ultwRTjzE_ps0aDqs1hAt39EzZL_ZovlWTAPR1EoYmGpO0ixEEvL8vQEMhYzqa71JClNe4sSMMUwv8gImsjknSbr_L5BuN5ob6Y6qolRgCM/GFBjXc/VHygjNQePyBdafg8RVASwgS3Yk6JrmhKm6BoduEk/k7J5f1aT3MC88w_Mvgc03oCBrmSqZ5tmTr330MWyUH505pFZZXE0EAtutXpqCNVCgea7cA7tpaWj4F/5IToO1/KyInpfKxYoDYrDGO56lYaiL33NCJM=',
    MD5OfBody: 'c81e728d9d4c2f636f067f89cc14862c',
    Body: '2',
    Attributes: {
    SenderId: 'AROAZ3AT2SL3K4UXABBDR:sqs_dlq_replay_function',
    ApproximateFirstReceiveTimestamp: '1622803955869',
    ApproximateReceiveCount: '6',
    SentTimestamp: '1622803940765',
    SequenceNumber: '18862182151616497664',
    MessageDeduplicationId: 'a19e29fe-92a1-47fa-895f-7839b7e05868',
    MessageGroupId: '3'
    }
    }

Is this an expected behavior? I'm using LocalStack and the MessageDeduplicationId is kept as-is after moving to the DLQ. I don't know AWS is correct, or LocalStack is correct?

Edited by: ducdnguyen on Jun 4, 2021 4:54 AM

asked 3 years ago957 views
3 Answers
0

Thank you for the report, we are looking into this behavior.

AWS
answered 3 years ago
0

We have confirmed that this is an intended, but not document behavior of the dead-leter queue (DLQ) mechanism when using SQS FIFO queues.

SQS supports DLQ mechanism, which sends an un-processed message to DLQ. Such process is very similar to SendMessage, so most of SendMessage constraints apply, too.

FIFO queues support deduplication id, to ensure exactly-once processing. We also use this field internally when sending a message to DLQ. As such, it needs to be a globally unique id, to make sure that the DLQ deduplication will not prevent storing of two independent messages that happen to share a deduplication id.

For example, both queue 1 and 2 are set to same DLQ. We enqueue message A to queue 1, and message B to queue 2. Both messages have same deduplication id. These messages may be sent to DLQ in the same time. If we keep the original deduplication id, only one of them will make it to DLQ - as the other will be recognized as duplicate.
To avoid this, SQS chooses to use message id, a unique id to identify them during such process, so both A and B can be enqueued into DLQ.

We will follow up with an update to the official documentation to explain this mechanism.

If you plan to use the original message deduplication id when consuming from DLQ, we suggest storing it as extra message attribute or as part of the message body.

AWS
answered 3 years ago
0

Thank you.
I had another thread going on with the AWS Support channel for this same behavior.
Just received the answer as well, and all answered my question.

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions