By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Queue solution to slow down speed rate of messages

0

Hello forum, I have a solutions architect question that need to be solved with some queuing strategy.

An external party sends us bursts of HTTP POST messages in a very quick rate of 15,000 per second. Our backend is not able to process all those messages at that speed. We need to receive them at 10 per second.

What would be the best option to slow down the speed of those messages?

Thank you for any feedback.

Topics
AnalyticsServerlessFront-End Web & MobileNetworking & Content DeliveryApplication Integration
Tags
Amazon Data FirehoseAmazon API GatewayAmazon Simple Queue ServiceAmazon Simple Notification Service (SNS)
Language
English
rePost-User-0583042
asked a year ago647 views
2 Answers
1

Hi, the expected answer is most likely an SQS queue, but if you can only process 10 per second those 15,000-message bursts better not happen more often than every 1,500 seconds! Also SQS messages are limited to 256KB each. And if processing order needs to be preserved or duplicates removed, it would need to be a FIFO SQS queue not a standard one with "at least once" delivery.

EXPERT
skinsman
answered a year ago
profile pictureAWS
EXPERT
kentrad
reviewed a year ago
  • rePost-User-0583042
    a year ago

    Thank you for your answer. No, those bursts don't happen more than once an hour and each message is only a few bytes.

0

The answer was actually easier than I originally thought. We just configured the receiving NGINX server to send 429 (Too many requests) after X messages/s.

rePost-User-0583042
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content