How does MSK Replicator work

0

if i am using MSK Replicator to replicate MSK topic data and consumer offset from cluster X to cluster Y and suddenly cluster X crashes, so i recreate a new cluster and i prefer use the same name X, how will the replicator work under this use case? and if i create a new MSK replicator to replicate data from cluster Y to cluster X, will this replicator copy data from Y to new cluster X?

And also does MSK replicator support source cluster is 2.8.2.tired and destination cluster is 3.6.0?

debug9
asked 3 months ago731 views
1 Answer
0

Hi, debug9

When using MSK Replicator to replicate MSK topic data and consumer offset from cluster X to cluster Y, and if cluster X crashes and you recreate a new cluster with the same name X, here's how the MSK Replicator would work:

Replicator Behavior for Recreated Cluster X:
    - If you recreate cluster X with the same name, MSK Replicator may not automatically resume replication to the new cluster X. MSK Replicator maintains a mapping between topics and consumer groups in the source and destination clusters. Recreating a cluster with the same name may not be automatically recognized as the same cluster by the replicator.
    - You may need to update the MSK Replicator configuration to point to the new cluster X, ensuring that the replication configuration is aligned with the new cluster's details.

Creating a New MSK Replicator to Replicate Data from Cluster Y to New Cluster X:
    - If you create a new MSK Replicator to replicate data from cluster Y to the new cluster X, the replicator will copy data from Y to the new cluster X as configured in the replication configuration.
    - This process is independent of the previous replication from old cluster X to cluster Y.

MSK Replicator Version Compatibility:
    - MSK Replicator's version compatibility depends on the versions of Apache Kafka running on the source and destination clusters.
    - From my understanding, it's recommended to use MSK Replicator with clusters running the same Kafka version to ensure compatibility and proper replication.

https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-msk-support-apache-kafka-version-3-6-0-tiered-storage/

profile picture
hoylem
answered 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions