How to measure the lag between AWS Redshift MV and MSK topics in an architecture involving AWS Redshift streaming ingestion from MSK?

0

In our architecture we have data from AWS MSK being ingested into AWS Redshift using the Amazon Redshift Streaming Ingestion feature. In this approach, we define an external schema where we have the Materialized Views (MVs) being defined on the topics in MSK cluster. These MVs have auto-refresh turned on.

At any given point in time, what are the design options to measure the lag between the data in the MSK topics and the Redshift Materialized Views i.e. how far the Redshift MVs are the from the kafka topics respectively?

  • Any inputs on this would be appreciated?

MK
asked 5 months ago113 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions