SQS - FIFO Queue - Exactly Once Processing

0

For FIFO Queue, If the visibility time-out expires while a consumer is still processing a message, will it become available and can be returned again and therefore could it be processed twice ? If not, how is it prevented to happen to ensure exactly once processing ?

DP
질문됨 4달 전262회 조회
3개 답변
1

Hello.

You may want to read the blog below.
If the visibility time-out is exceeded, messages from other consumers can be processed, so in order to avoid processing those messages, add a process to the consumer that writes the message to another storage and checks whether it is duplicated.
https://sookocheff.com/post/messaging/dissecting-sqs-fifo-queues/

Concurrent Consumers: Visibility Timeout Error

The problem is even worse than that: even with a single consumer, if you process a message and then fail to delete the message from SQS within the visibility timeout, the message will be delivered twice. One solution to guard against this would be for the consuming system to read messages from SQS into their own durable storage for future processing. After confirming that a copy of the message has been durably committed, the consumer can delete the message from SQS. This concept is similar to how many database systems use a write-ahead-log to durably store commits. This scenario is depicted in the figure that follows: after each message is committed to the database, it is deleted from the SQS queue, while in parallel a message processor running as a background thread handles processing the message.

profile picture
전문가
답변함 4달 전
1

It uses a message deduplication ID https://docs.aws.amazon.com/sns/latest/dg/fifo-message-dedup.html

If a message with a particular deduplication ID is successfully published to an Amazon SNS FIFO topic, any message published with the same deduplication ID, within the five-minute deduplication interval, is accepted but not delivered. The Amazon SNS FIFO topic continues to track the message deduplication ID, even after the message is delivered to subscribed endpoints.

profile picture
전문가
Steve_M
답변함 4달 전
1

SQS FIFO guarantees that you will get each message once, only if you process it and delete it during the visibility timeout. If you do not delete the message or extend the visibility timeout, the message will be available for other consumers.

This is different than standard queue, where two consumers may get the same message, even if the visibility timeout did not expire yet.

profile pictureAWS
전문가
Uri
답변함 4달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠