Stack Kafka Message Monitoring

Question

I am replicating data from on-premises to AWS. The first component is a Mirror Maker 2.0 that passes the messages to a MSK Topic and that passes it to a MSK Connector that in turn inserts it into the Aurora Postgress database. How can I monitor the message from the Mirror Maker to the MSK Connector? Or until the record is inserted into Aurora? I would like metrics on message delay between components, error message, total input and output bytes. Is it possible?

Ed Berezitsky · Answer

End-to-end latency for the flow described won't be support out the box.

You can monitor MM2 metrics:
```
    record-age-ms                          # age of records when they are replicated
    record-age-ms-min
    record-age-ms-max
    record-age-ms-avg
    replication-latecny-ms                 # time it takes records to propagate source->target
    replication-latency-ms-min
    replication-latency-ms-max
    replication-latency-ms-avg

```
Second part of you flow, when you use MSK/Kafka Connect to send records to Aurora, you can monitor using consumer group lag metrics that are available in MSK Cloudwatch/JMX metrics, as a sink connector will be using regular kafka consumer group to get data from a topic.

This definitely won't show a latency on individual message level. In order to address this, you need to implement some kind of data lineage. This probably can be achieved with interceptors and metadata on messages' header with timestamps when message is passing through MM2 and Kafka Connect.

Paulo · Answer

If my understanding is correct, these metrics are available in Mirror Maker and monitor up to the next component of the architecture which is the MSK Topic and not up to the MSK connector. Is that right?

Stack Kafka Message Monitoring

Add your answer

Relevant content