SQS - FIFO Queue - Exactly Once Processing

0

For FIFO Queue, If the visibility time-out expires while a consumer is still processing a message, will it become available and can be returned again and therefore could it be processed twice ? If not, how is it prevented to happen to ensure exactly once processing ?

DP
preguntada hace 4 meses261 visualizaciones
3 Respuestas
1

Hello.

You may want to read the blog below.
If the visibility time-out is exceeded, messages from other consumers can be processed, so in order to avoid processing those messages, add a process to the consumer that writes the message to another storage and checks whether it is duplicated.
https://sookocheff.com/post/messaging/dissecting-sqs-fifo-queues/

Concurrent Consumers: Visibility Timeout Error

The problem is even worse than that: even with a single consumer, if you process a message and then fail to delete the message from SQS within the visibility timeout, the message will be delivered twice. One solution to guard against this would be for the consuming system to read messages from SQS into their own durable storage for future processing. After confirming that a copy of the message has been durably committed, the consumer can delete the message from SQS. This concept is similar to how many database systems use a write-ahead-log to durably store commits. This scenario is depicted in the figure that follows: after each message is committed to the database, it is deleted from the SQS queue, while in parallel a message processor running as a background thread handles processing the message.

profile picture
EXPERTO
respondido hace 4 meses
1

It uses a message deduplication ID https://docs.aws.amazon.com/sns/latest/dg/fifo-message-dedup.html

If a message with a particular deduplication ID is successfully published to an Amazon SNS FIFO topic, any message published with the same deduplication ID, within the five-minute deduplication interval, is accepted but not delivered. The Amazon SNS FIFO topic continues to track the message deduplication ID, even after the message is delivered to subscribed endpoints.

profile picture
EXPERTO
Steve_M
respondido hace 4 meses
1

SQS FIFO guarantees that you will get each message once, only if you process it and delete it during the visibility timeout. If you do not delete the message or extend the visibility timeout, the message will be available for other consumers.

This is different than standard queue, where two consumers may get the same message, even if the visibility timeout did not expire yet.

profile pictureAWS
EXPERTO
Uri
respondido hace 4 meses

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas