Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

AWS MSK - Capture change of events

0

Hello,

We have a producer (Lambda) and a consumer setup (ECS tasks) with a MSK (Kafka) as the queue service.

Let's say a producer (lambda in our case) puts a message on the broker and consumer consumes it. Is there a way to log the message (including it's payload) and the change of events that occurred for that message before being picked up by the consumer?

Basically, I'd like to monitor if the message was put in the queue and the time it was consumed from the queue.

I know MSK does not provide such support but I want to check if there is a way to to do using SQS/Lambda/Kinesis.

Thank you

Topics
Application IntegrationServerlessAnalyticsManagement & GovernanceComputeDeveloper Tools
Tags
Amazon Managed Streaming for Apache Kafka (Amazon MSK)AWS LambdaMonitoring & LoggingAmazon Simple Queue Service
Language
English
asked a year ago135 views
1 Answer
0

Hi,

I would suggest that you MSK Replicator to be able to analyze all the traffic from the replica without disturbing the application consuming the traffic from the original MSK

See https://aws.amazon.com/blogs/aws/introducing-amazon-msk-replicator-fully-managed-replication-across-msk-clusters-in-same-or-different-aws-regions/

Enter image description here

Best,

Didier

EXPERT
answered a year ago
  • SidK
    a year ago

    If I understand correctly, selecting "detect and copy new topics" will replicate the messages to the Target MSK Cluster. How can I differentiate between a message put in the topic vs if it has been ack'd (or consumed)?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content