AWS MSK connector for datagen source is failing without errors

Question

Hi ,

We have a msk cluster provisioned and try to create custom connector datagen source for s3, but its failing without errors, we can only see below logs in broker logs:

[2022-08-26 12:52:00,713] ERROR [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=1] Error for partition __amazon_msk_connect_offsets_S3-Sink-Orders_ae585cdd-321c-4e70-8de0-a115ed0e127b-4-13 at offset 0 (kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
[2022-08-26 12:52:00,713] ERROR [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=1] Error for partition __amazon_msk_connect_offsets_S3-Sink-Orders_ae585cdd-321c-4e70-8de0-a115ed0e127b-4-19 at offset 0 (kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
[2022-08-26 12:52:00,713] ERROR [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=1] Error for partition __amazon_msk_connect_offsets_S3-Sink-Orders_ae585cdd-321c-4e70-8de0-a115ed0e127b-4-7 at offset 0 (kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
[2022-08-26 12:52:00,713] ERROR [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=1] Error for partition __amazon_msk_connect_offsets_S3-Sink-Orders_ae585cdd-321c-4e70-8de0-a115ed0e127b-4-1 at offset 0 (kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.

Answer

Well, there could be various reasons for this issue:

However, the main issue is that your connector is not able to fetch the connector offset topic __amazon_msk_connect_offsets_S3-Sink-Orders_ae585cdd-321c-4e70-8de0-a115ed0e127b-4-1 or your cluster doesn’t have that topic at all.

Make sure that you have given proper permissions(IAM role) to your connector to create this topic as per the below documentation:

[+] https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-service-execution-role.html

If the issue still persists, please reach out to us over a support case.

AWS MSK connector for datagen source is failing without errors

Relevanter Inhalt