By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

inconsistency of SQS (Simple Queue Service) message reception from SNS (Simple Notification Service) fan-out

0

We are currently experiencing a critical issue in our production environment related to the inconsistency of SQS (Simple Queue Service) message reception from SNS (Simple Notification Service) fan-out. Our setup involves an SNS topic triggering SQS queues based on message attributes. While the email delivery mechanism triggered by the same SNS topic and message attribute is functioning properly, the SQS queues intermittently fail to receive messages. This inconsistency poses a significant operational risk to our system.

Topics
ServerlessApplication Integration
Tags
Amazon Simple Queue ServiceAmazon Simple Notification Service (SNS)
Language
English
Vijay Anbu
asked 2 months ago175 views
2 Answers
1
Accepted Answer

How can you tell that the messages do not arrive in the queue? Did you check the CloudWatch metrics for the queue for the number of sent messages?

Could it be that you have more than one consumer on the queue? You should only have one as the consumer deletes the message from the queue after processing it. If you have more than one consumer, each one will get a subset of the messages.

profile pictureAWS
EXPERT
Uri
answered 2 months ago
profile picture
EXPERT
Oleksii Bebych
reviewed a month ago
  • Vijay Anbu
    2 months ago

    Hello Uri,

    Yes, that's right. We have more than one consumer accidentally configured for the same SQS. That's the root cause of the issue. Thank you so much for your answer.

    Regards, Vijay

0

Are you able to take the messages that failed to be received by SQS and resend them? That would allow you to confirm if the queue just failed to receive it or if the filter subscription filter policy is why the queue isn't receiving the messages. Also, do you have a DLQ to see if there was a problem processing the message by SQS. The SNS integration with SQS will retry any failed message attempts.

AWS
matthew_d
answered 2 months ago
  • Vijay Anbu
    2 months ago

    Hi Matthew,

    Yes, I attempted to send the same message, and it eventually worked after a few attempts. I don't see any issues with the filter policy. Another subscription for email, using the same filter policy, consistently works fine.

    Additionally, I have configured a Dead Letter Queue (DLQ) for the SQS, but there are no messages in it.

    The current SNS settings include 3 retries. I attempted to disable encryption, but it still did not resolve the issue. Interestingly, the same settings in my lower environments are functioning correctly; they have not failed even once.

  • matthew_d
    2 months ago

    SNS's integration with SQS will retry 3 times immediately but then starts using backoff and will make over 100k attempts over 23 days. I agree the filter policy isn't the problem if sending the same message eventually goes through and it works in your lower environments.

    How are you confirming that SQS is processing the message? Are you using CloudWatch NumberOfMessagesSent metric to check that a message was received by SQS from SNS?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content