AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.
AWS MSK is not able to load balance records to all the consumers in a consumer group
Could anyone help me out? I have stuck on load balances the kafka records to my consumers
This is my MSK set up
Each zone has 1 broker
Each topic has 3 partition
The producer is my IoT core rule as you see I have used the thing name as key and use the mod function to obtain partition from 0, 1 ,2
The consumer is my 3 ec2 instance which I am using javascript and used the kafkajs library to subscribe the topic.
I expected the MSK can evenly distribute the message to all 3 ec2 instances like using round robin For example Iot core send message to partition 1 and consumer( EC2 A) consumes the records, and then Iot core send message to partition 2 and consumer(EC2 B) consume the records.
but currently, all the records is sending to EC2 A and the other 2 EC2 remain idle...
this is my js code
this is the log console
as you see on the left side (EC2 B) always remains idle after connect to kafka and on the right side (EC2 A) keeps consuming records Also, (EC2 C) is always idle...
- Newest
- Most votes
- Most comments
The issue you're experiencing with Amazon MSK not load balancing records to all consumers in your consumer group is likely due to how partitions are assigned in Apache Kafka. Here's what's happening and how you can address it:
-
Partition Assignment: In Kafka, partitions are the unit of parallelism. Each partition can only be consumed by one consumer within a consumer group at a time. With your current setup of 3 partitions and 3 consumers, you should ideally see each consumer processing messages from one partition.
-
Single Partition Usage: Based on your producer configuration, it appears that you're using a mod function to determine the partition. This might be causing all your messages to be sent to a single partition, which is why only one consumer (EC2 A) is receiving all the records while the others remain idle.
-
Consumer Group Behavior: When multiple consumers are in the same group, Kafka assigns partitions to consumers. If all messages are going to one partition, only one consumer will be active.
To resolve this and achieve better load balancing:
-
Ensure Even Distribution: Modify your IoT Core rule to distribute messages more evenly across all partitions. Instead of using a mod function, consider using a round-robin approach or a hash of the key that spreads messages across all partitions.
-
Verify Consumer Group: Make sure all your consumers are part of the same consumer group. This allows Kafka to distribute partitions among them.
-
Check Partition Count: Ensure you have at least as many partitions as consumers. In your case, 3 partitions for 3 consumers is correct.
-
Monitor Partition Assignment: Use Kafka monitoring tools to verify that partitions are being assigned to different consumers.
-
Consider Increasing Partitions: If you expect to scale beyond 3 consumers, consider increasing the number of partitions. This allows for better parallelism as your system grows.
By implementing these changes, particularly focusing on how messages are distributed to partitions from your IoT Core rule, you should see a more even distribution of messages across your consumers.
Sources
Community | Apache Kafka Partitions as a Unit of Parallelism
Community | In the land of the sizing, the one-partition Kafka topic is king
Relevant content
asked a year ago
asked 3 years ago
- AWS OFFICIALUpdated a year ago
