Copying data from IBM MQ to Amazon MSK

0

I am trying to use the Kafka Connect source connector for copying data from IBM MQ into Amazon MSK. I was able to create the custom plugin in MSK, but when creating the connector it is transitioning to failed state.
The custom plugin i build from. https://github.com/ibm-messaging/kafka-connect-mq-source
For the connector configuration i am using
connector.class=com.ibm.eventstreams.connect.mqsource.MQSourceConnector
mq.connection.name.list=<host>(1414)
mq.channel=<channel name>
tasks.max=1
mq.queue.manager=<name>
mq.queue=<queue name>
mq.channel.name=<channel name>
kafka.topic=<topic name>
value.converter=org.apache.kafka.connect.storage.StringConverter
key.converter=org.apache.kafka.connect.storage.StringConverter
mq.connection.mode=client
The configuration works and i am able to copy messages from IBM MQ to Apache Kafka installed on EC2. It is not working from IBM MQ to Amazon MSK. Are there any limitations on what source connectors you use with Amazon MSK?

asked 2 years ago1046 views
2 Answers
0

You can use most connectors with MSK Connect, as long as they do not need to actively listen themselves. In the case of the MQ connector, this should work for you. The issue is most likely connected to the networking setup and the ability of the connector to find your MQ instances. Since networking can be very dependent on your environment it is difficult to be specific, and you may be able to get more details by sharing the logs and specifics with the AWS support team. However, some common challenges include name resolution, and making sure your host names are resolvable from the connector's perspective.

answered 2 years ago
0

Thank you for your response. I don't believe it is a networking issue as i tried the reachability analyzer from the VPC end point where the IBM MQ is installed and the AWS Kafka Connect ENI and it is reachable. I also enabled the cloudwatch log monitoring and don't see any errors in the logs. So i am not sure if there is an issue with the connector configuration and if the properties specified are correct as i don't see I also don't see any message saying connectivity is established.

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions