Cloudwatch metric evaluation - aws msk kafka

Question

I have 8 AWS MSK Kafka clusters. I thought of merging them into one and moving to serverless to reduce cost. But MSK Kafka serverless have some limitations/quota. I have to ensure that after merging the clusters the total workload comes under the limit/quota - So what cloudwacth metrices should i evaluate for each of the following to get the corrrect data related to my workload

https://docs.aws.amazon.com/msk/latest/developerguide/limits.html

How to find the each of these dimension for my current workload (i have 8 clusters with different number of brokers) 3.retention duration 4.number of client connections 5.connection attempts 6. message size 7.request rate 8. rate of topic management APIs requests rate 9.fetch bytes per request 10.number of consumer groups 12.rate of partition creation and deletion 13.ingress throughput per partition 14.egress throughput per partition 15.partition size (for compacted topics) 16.number of client VPCs per serverless cluster

AWS-Repost-User · Answer

Other details shared on the [same question](https://repost.aws/questions/QU81j_MtR8S4CMcp3EYCJpWg/merging-multiple-msk-kafka-clusters)

AWS-Repost-User · Answer

Check this [blog](https://aws.amazon.com/blogs/big-data/how-to-choose-the-right-amazon-msk-cluster-type-for-you/) which guides on how to choose MSK cluster type depending on your use case. Noting that Serverless requires IAM authentication, which may or may not be applicable for your case.

One of the plans is that you may start merging all, one after one, into a provisioned cluster. Once all are working on one cluster, then you can study the migration of this cluster into a Serverless cluster based on all the requirements and specs.

Cloudwatch metric evaluation - aws msk kafka

Contenuto pertinente