MM2 replication.policy.class doesn't find JAR for MSK Connect Connector

0

Goal: would like to use the connector config property replication.policy.class to configure a custom replication policy to have topics names be the same in active/passive MirrorMaker2 replication setup that runs on MSK connect. But I can't get the connector config to recognize the custom JAR. I suspect the JAR isn't being copied to the right location on the VM where MM is running.

My expectation is that if I place the JAR in the zip file that is required for the plugin that it will be copied to the right location for MM to pick it up.

Steps to reproduce (console):

  1. Clone https://github.com/aws-samples/mirrormaker2-msk-migration.git and compile the CustomMM2ReplicationPolicy class into a jar (mvn clean install).
  2. Put this jar into a zip and put it in s3.
  3. Create custom plugin using this zip that's in s3.
  4. Create a connector using the following config:
connector.class=org.apache.kafka.connect.mirror.MirrorSourceConnector
replication.factor=2
offset-syncs.topic.replication.factor=1
consumer.group.id=mm2-msc
topics=.*
tasks.max=3
sync.topic.acls.interval.seconds=20
producer.enable.idempotence=true
source.cluster.bootstrap.servers=<ANY MSK KAFKA BS>
sync.topic.configs.interval.seconds=20
target.cluster.security.protocol=PLAINTEXT
target.cluster.bootstrap.servers=<ANY MSK KAFKA BS>
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
clusters=prim, sec
source.cluster.security.protocol=PLAINTEXT
refresh.groups.interval.seconds=20
replication.policy.class=com.amazonaws.kafka.samples.CustomMM2ReplicationPolicy

Expected result: No error and topics are created in target with same name as source.

Actual result: CustomMM2ReplicationPolicy could not be found.

I suspect the reason that it fails is that the jar is not copied where it should be so that it can be picked up (the libs folder).

I can work around this by setting the following to empty values:

replication.policy.separator= 
source.cluster.alias= 
target.cluster.alias= 

Others seem to have had success with this: https://stackoverflow.com/questions/59390555/is-it-possible-to-replicate-kafka-topics-without-alias-prefix-with-mirrormaker2. Can I assume this will safely work in MSK connect? Is there some other preferred way to work around this issue when using MSK connect to host MM?

asked a year ago245 views
1 Answer
0

Dear AWS Customer

Following response has been posted on re:Post for the customer query:

Please note that, by default, MM2 creates destination topics with the source cluster alias as prefix. For example, assuming the source cluster alias is "primary", topics in the target cluster will be named as "primary.<topic_name> ".

The error you are getting "CustomMM2ReplicationPolicy could not be found" can be due to various reasons. In order to troubleshoot this further, we require details that are non-public information. Please open a support case with AWS using the following link:

https://console.aws.amazon.com/support/home#/case/create

AWS
SUPPORT ENGINEER
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions