By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Lenses.io source connector plugin in MSK connect not flushing S3 files to Kafka topic

0

I am using AWS MSK connect to send files from S3 bucket to Kafka topic. For this, I am using lenses.io source connector plugin: https://docs.lenses.io/5.0/integrations/connectors/stream-reactor/sources/s3sourceconnector/ The connector is streaming the existing files in S3 to Kafka right after the creation. However, when more files are added in S3, the source connector is not streaming those files to Kafka topic any longer.

Can someone help if I am missing anything critical on the configuration side? I can't find any documentation for S3 source connector in AWS MSK documentation. I could just get the documentation for S3 sink connector: https://docs.aws.amazon.com/msk/latest/developerguide/mkc-create-connector.html. Wondering if source connector can be used for production or not?

Below is the configuration for the connector

Source connector configuration:

connector.class=io.lenses.streamreactor.connect.aws.s3.source.S3SourceConnector connect.s3.kcql=insert into test_topic select * from test_bucket:tutorial STOREAS json tasks.max=1 topics=test_topic value.converter=org.apache.kafka.connect.storage.StringConverter errors.log.enable=true key.converter=org.apache.kafka.connect.storage.StringConverter

Topics
Analytics
Tags
Amazon Managed Streaming for Apache Kafka (Amazon MSK)
Language
English
rePost-User-8610805
asked a year ago323 views
2 Answers
0

Hello there,

There could be a reason that MSK Connector is failing due to some issue. You can check the connector logs to confirm if there are any errors or not. Also, you can check if there are any errored tasks from CloudWatch metric monitoring. However, it needs to be investigated from our side to confirm the issue.

Could you please reach back to us over a Support case with your MSK Connector ARN and the CloudWatch logs if possible.

AWS
SUPPORT ENGINEER
Mahesh_S
answered 7 months ago
0

There have been many updates to the S3 Connector in the last 6 months. Please ensure you are using the latest build.

For problems with the Lenses S3 (or any other) Connector, please check out the Lenses forum at ask.lenses.io or you can speak with the team directly on our Community Slack (register via https://launchpass.com/lensesio)

Gui
answered 4 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content