Lenses.io source connector plugin in MSK connect not flushing S3 files to Kafka topic

Question

I am using AWS MSK connect to send files from S3 bucket to Kafka topic. For this, I am using lenses.io source connector plugin: https://docs.lenses.io/5.0/integrations/connectors/stream-reactor/sources/s3sourceconnector/
The connector is streaming the existing files in S3 to Kafka right after the creation. However, when more files are added in S3, the source connector is not streaming those files to Kafka topic any longer.

Can someone help if I am missing anything critical on the configuration side? I can't find any documentation for S3 source connector in AWS MSK documentation. I could just get the documentation for S3 sink connector: https://docs.aws.amazon.com/msk/latest/developerguide/mkc-create-connector.html. Wondering if source connector can be used for production or not?

Below is the configuration for the connector

**Source connector configuration:**

connector.class=io.lenses.streamreactor.connect.aws.s3.source.S3SourceConnector
connect.s3.kcql=insert into test_topic select * from test_bucket:tutorial STOREAS json
tasks.max=1
topics=test_topic
value.converter=org.apache.kafka.connect.storage.StringConverter
errors.log.enable=true
key.converter=org.apache.kafka.connect.storage.StringConverter

Answer

There have been many updates to the S3 Connector in the last 6 months.  Please ensure you are using the latest build.

For problems with the Lenses S3 (or any other) Connector, please check out the Lenses forum at [ask.lenses.io]() or you can speak with the team directly on our Community Slack (register via https://launchpass.com/lensesio)

Answer

Hello there,

There could be a reason that MSK Connector is failing due to some issue. You can check the connector logs to confirm if there are any errors or not. Also, you can check if there are any errored tasks from CloudWatch metric monitoring.  However, it needs to be investigated from our side to confirm the issue.

Could you please reach back to us over a Support case with your MSK Connector ARN and the CloudWatch logs if possible.

Lenses.io source connector plugin in MSK connect not flushing S3 files to Kafka topic

Relevanter Inhalt