Choosing between kafka and sqs as iot rule

0

I am building a platform using aws iot core. Here, Device will publish the message to mqtt and then I want to use iot rule to route it to either sqs or kafka, so that consumer will consume message from the queue/topic. I am confused between sqs and kafka, and need some clarity on differences between both. It would be helpful if I get the comparison on below aspects:

  1. Scalable
  2. Performance
  3. Cost effective
  4. Reliable
  5. Implementation/Management complexity
2 Answers
0

While SQS is a message queue service that is fully managed, Kafka, in contrast, is a self-managed platform that serves as a distributed streaming platform, enabling the real-time processing of streaming data. If you require a message queue service that is fully managed and offers both standard and FIFO delivery options, SQS might be the suitable option. Conversely, if you require a distributed streaming platform capable of storing and processing real-time data, Kafka may be a more suitable choice.

profile pictureAWS
EXPERT
answered 10 months ago
0
  1. Scalable - SQS scales automatically to your needs without you needing to do anything. In Kafka you will need to set the cluster size, and changing it might be challenging.
  2. Performance - What do you mean by that in this case? Throughput? SQS can handle any load, Kafka depends on the cluster size. Latency? You will need to test to find out which is better in your use case.
  3. Cost effectiveness - In SQS you pay by the number of API calls that you make (which correlates to the number of messages you send), in Kafka you pay by the cluster size, regardless of the number of messages. In most cases I think that SQS will cost less, but ut may be that in very high numbers Kafka will be more cost effective. You will need to do the math for your use case.
  4. Reliable - Both of them are very reliable services used by many users.
  5. SQS has no management complexity. You just create the queue and start using it with the different SDKs. Kafka on the other hand you will need to create a cluster, guess it's sizing, manage it, etc. Much more complex.

I would say that in most cases, SQS will be the better option, unless you have some specific requirements.

profile pictureAWS
EXPERT
Uri
answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions