Visibility of User-Created Consumer Groups and Topics in AWS MSK CloudWatch Metrics

The situation you're describing highlights some nuances in how Amazon MSK (Managed Streaming for Apache Kafka) metrics are reported in CloudWatch.

The SumOffsetLag metric is indeed included in the Default Level monitoring, but its visibility for user-created consumer groups and topics can be limited. The default metrics you're seeing (amazon.msk.canary.group.broker-1, amazon.msk.canary.group.broker-2, and __amazon_msk_canary) are internal MSK metrics used for system health monitoring.

To gain visibility into user-created consumer groups and topics, you need to enable more detailed monitoring levels. The PER_TOPIC_PER_BROKER or PER_TOPIC_PER_PARTITION monitoring levels provide metrics for user-created topics and consumer groups.

The inclusion of SumOffsetLag in Default Level monitoring might be to provide a baseline metric for overall cluster health and performance, even if it doesn't give granular insights into user-specific data at that level.

Regarding your question about enabling Enhanced Topic Level monitoring for only a few topics: Unfortunately, Amazon MSK doesn't currently offer a way to enable detailed monitoring for specific topics only. When you upgrade the monitoring level, it applies to all topics in the cluster. This can indeed lead to unnecessary costs if you only need to monitor a few topics.

To mitigate this, you might consider:

1. Using application-level monitoring for the specific topics you're interested in.
2. Implementing custom metrics collection for your critical topics.
3. Using CloudWatch Logs Insights or other log analysis tools to extract relevant information about specific topics from broker logs.

While these approaches may require more setup, they can help you achieve more targeted monitoring without incurring the full cost of cluster-wide enhanced monitoring.

It's worth noting that AWS continually updates its services based on customer feedback. If selective topic monitoring is important to you, consider providing this feedback to AWS support or through their feature request channels.
Sources
Basic monitoring and detailed monitoring in CloudWatch - Amazon CloudWatch
Amazon MSK metrics for monitoring with CloudWatch - Amazon Managed Streaming for Apache Kafka

Hello,

Default/Basic level of monitoring DOES cover user-created consumers and user-created topic. Here is what I can see in my CloudWatch:

I run my consumer: kafka/bin/kafka-console-consumer.sh --topic sampletopic --bootstrap-server $b --group custom-group-1 --consumer.config ~/environment/config/client_iam.properties --from-beginning

This is what I have in CW:

Note, It is possible though, that default naming of console consumers, like console-consumer-*** will be excluded from the metrics. At least I don't see them in CW.