SQS Fifo TPS Limit and Solution

Question

Hi,
I am building a service which need a queue for sending message from one service to another service. For this i need a FIFO queue but FIFO queue has a API call limit of 3000/sec and with batch of 10 message i can maximum get 30000 message per second. But my Application need more API Call limit as i need around 50k to 60k message per second.

Is this possible by any way that i can achieve this limit with SQS FIFO.
Or is there any other solution for this problem?

Answer

You can look at using Kinesis Data Streams (https://aws.amazon.com/kinesis/data-streams/) or Managed Kafka Service (https://aws.amazon.com/msk/)

Kinesis maintains the ordering of messages per shard and Kafka maintains the ordering of messages per partition

Answer

If you are using Lambda as the consumer, you can just create multiple queues, all triggering the same function. Distribute the messages between the queues based on some group ID hashing algorithm, so that all messages with the same group ID will get to the same queue.

SQS Fifo TPS Limit and Solution

Contenuto pertinente