Goal: would like to use the connector config property replication.policy.class
to configure a custom replication policy to have topics names be the same in active/passive MirrorMaker2 replication setup that runs on MSK connect. But I can't get the connector config to recognize the custom JAR. I suspect the JAR isn't being copied to the right location on the VM where MM is running.
My expectation is that if I place the JAR in the zip file that is required for the plugin that it will be copied to the right location for MM to pick it up.
Steps to reproduce (console):
- Clone https://github.com/aws-samples/mirrormaker2-msk-migration.git and compile the CustomMM2ReplicationPolicy class into a jar (
mvn clean install
).
- Put this jar into a zip and put it in s3.
- Create custom plugin using this zip that's in s3.
- Create a connector using the following config:
connector.class=org.apache.kafka.connect.mirror.MirrorSourceConnector
replication.factor=2
offset-syncs.topic.replication.factor=1
consumer.group.id=mm2-msc
topics=.*
tasks.max=3
sync.topic.acls.interval.seconds=20
producer.enable.idempotence=true
source.cluster.bootstrap.servers=<ANY MSK KAFKA BS>
sync.topic.configs.interval.seconds=20
target.cluster.security.protocol=PLAINTEXT
target.cluster.bootstrap.servers=<ANY MSK KAFKA BS>
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
clusters=prim, sec
source.cluster.security.protocol=PLAINTEXT
refresh.groups.interval.seconds=20
replication.policy.class=com.amazonaws.kafka.samples.CustomMM2ReplicationPolicy
Expected result: No error and topics are created in target with same name as source.
Actual result: CustomMM2ReplicationPolicy could not be found
.
I suspect the reason that it fails is that the jar is not copied where it should be so that it can be picked up (the libs
folder).
I can work around this by setting the following to empty values:
replication.policy.separator=
source.cluster.alias=
target.cluster.alias=
Others seem to have had success with this: https://stackoverflow.com/questions/59390555/is-it-possible-to-replicate-kafka-topics-without-alias-prefix-with-mirrormaker2. Can I assume this will safely work in MSK connect? Is there some other preferred way to work around this issue when using MSK connect to host MM?
Ok Norman I created a case. Case number 12977404621.