Merging multiple MSK Kafka Clusters

0

I have 8 AWS MSK Kafka clusters. I thought of merging them into one and moving to serverless to reduce cost. But MSK Kafka serverless have some limitations/quota. I have to ensure that after merging the clusters the total workload comes under the limit/quota - https://docs.aws.amazon.com/msk/latest/developerguide/limits.html How to find the each of these dimension for my current workload (i have 8 clusters with different number of brokers) 3.retention duration 4.number of client connections 5.connection attempts 6. message size 7.request rate 8. rate of topic management APIs requests rate 9.fetch bytes per request 10.number of consumer groups 12.rate of partition creation and deletion 13.ingress throughput per partition 14.egress throughput per partition 15.partition size (for compacted topics) 16.number of client VPCs per serverless cluster

https://docs.aws.amazon.com/msk/latest/developerguide/limits.html

asked 3 months ago117 views
1 Answer
0

I think that you have to take these metrics from Cloudwatch little by little. Then with the MSK Cluster Sizer you can really see what yuu can achieve with the new confifguration.

https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html

profile picture
EXPERT
answered 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions