Cluster stopped publishing metrics

0

Hi,
my cluster has 3 broker and Prometheus monitoring is enabled. Everyhing was fine and this afternoon 2 of 3 brokers stopped publishing metrics.

curl -s b-1.XXX.kafka.eu-central-1.amazonaws.com:11001/metrics | grep kafka_consumer_group_ConsumerLagMetrics_Value

Any ideas why this metric for example is missing on 2 brokers?

asked 3 years ago422 views
3 Answers
0

This sounds like a temporary problem specific to these brokers, that may occur during maintenance. The best way to address issues like this is usually though a support ticket where we can access and discuss more specific details of you cluster. Please do raise a ticket for us with the details of your cluster, and we will take a look.

answered 3 years ago
0

After 3 day the metrics are back.

answered 3 years ago
0

It is worth noting that if you have very high partition counts, or are using smaller broker instance types, especially the t3 nodes which are intended primarily to development use, then there can be resource constraints that affect the publication of metrics like this. It is worth raising a support ticket to look into the specific details of the case, but the most common cause of these events is brokers that are small for their workload.

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions