Maximum number of partitions in KRaft MSK

0

I already noticed Best practices for Standard brokers(https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html). But I think the post related to Zookeeper not KRaft cluster. I know that the KRaft cluster more effiecient to manage metadata than Zookeeper, so may be maximum number of partitions is higher. Could you inform the answer of the question? Thank you.

asked 2 months ago72 views
1 Answer
0

You're correct that the information about partition limits in the best practices for Standard brokers is related to ZooKeeper-based clusters, not KRaft clusters. KRaft (Kafka Raft) mode is indeed more efficient in managing metadata than ZooKeeper.

While there isn't specific information provided about the maximum number of partitions for KRaft clusters in Amazon MSK, we can infer that the limit is likely higher than for ZooKeeper-based clusters. This is because KRaft mode allows for more brokers per cluster without requiring a limit increase.

In KRaft mode, Amazon MSK enables the creation of clusters with up to 60 brokers, compared to the default quota of 30 brokers in ZooKeeper mode. This increase in the number of brokers suggests that KRaft clusters can support a higher number of partitions overall.

However, it's important to note that the exact maximum number of partitions for KRaft clusters is not explicitly stated in the available information. The cluster-wide partition limit still defines the upper bounds of scalability within the Kafka system, as it determines the maximum number of partitions that can be distributed across the available nodes.

If you need a specific number for the maximum partitions in a KRaft MSK cluster, it would be best to contact AWS support directly for the most up-to-date and accurate information. They would be able to provide you with the exact limits for KRaft clusters in Amazon MSK.
Sources
Introducing support for Apache Kafka on Raft mode (KRaft) with Amazon MSK clusters | AWS Big Data Blog
Amazon MSK launches support for KRaft mode for new Apache Kafka clusters - AWS

profile picture
answered 2 months ago
profile picture
EXPERT
reviewed 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions