Using MSK connect with a confluent kafka cluster - passing sasl configuration to the custom worker configuration (or is there another way to do this)

Question

I need to run a connector in MSK connect that connects to existing offsets topic in confluent kafka 
confluent kafka clusters have the following configuration to connect:

ssl.endpoint.identification.algorithm=https
 security.protocol=SASL_SSL
 sasl.mechanism=PLAIN
 sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="{{ CLUSTER_API_KEY }}" password="{{ CLUSTER_API_SECRET }}"; 
(Where the templated stuff is replaced by the real api/secret)
But trying to pass those via custom worker configuration fails as it doesn't accept those params (they are explicitly blocked)
Is there any other way to pass those? a prefix missing (tried cluster and cluster.target but nope)

Ed Berezitsky · Answer

Currently, MSK Connect supports no-auth or IAM only. You won't be able to provide required worker configuration.

baitanik · Answer

My Msk cluster is up & running with IAM policy.. 
while connecting to msk from local, need to provide

```
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
```

now when we are trying to create a connector , in Configuration settings , we are adding the above configs .. 
but still getting error : 
INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager:235)
org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1746775156865, tries=1, nextAllowedTryMs=1746775156966) timed out at 1746775156866 after 1 attempt

msk & connector are in same vpc... could you give some pointer how to fix it.

```
connector.class=io.questdb.kafka.QuestDBSinkConnector
topics=test2
tasks.max=10
include.key=false
client.conf.string=http::addr=:9000;username=admin;password=123;
name=connector2questdb4
value.converter.schemas.enable=false
table=dtest_kf
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
```

could someone give some idea.. how to resolve ?

Using MSK connect with a confluent kafka cluster - passing sasl configuration to the custom worker configuration (or is there another way to do this)

Add your answer

Relevant content