Kafka Connect Offset Commit Error on AWS MSK

0

I'm encountering an issue with Kafka Connect deployed on AWS Managed Streaming for Apache Kafka (MSK). Specifically, I'm observing an error related to offset committing when using Kafka Connect sink connectors.

Error Message:

2024-02-12 01:11:26,142 ERROR || WorkerSinkTask{id=XXXXX.public.XXXXX2-0} Commit of offsets threw an unexpected exception for sequence number 2506: {XXXXX.public.XXXXX-0=OffsetAndMetadata{offset=437542, leaderEpoch=null, metadata=''}}

Context:

I have Kafka Connect deployed on AWS MSK to stream data into various external systems. The error occurs intermittently during offset committing for sink tasks. Offset committing is critical for ensuring data consistency and fault tolerance in my data pipeline. What I've Tried:

Reviewed Kafka Connect logs, but the error message provides limited insight into the root cause. Verified the configuration settings for Kafka Connect and the sink connectors, ensuring alignment with best practices. Consulted Kafka documentation and online forums for troubleshooting guidance but haven't found a specific solution for this error. Attempted to restart Kafka Connect, but the error persists even after service restarts.

Expectations: I expected offset commits to be performed without errors, ensuring the reliability of my Kafka Connect setup. Given the importance of offset committing for data consistency, I'm seeking guidance on resolving this issue effectively.

질문됨 3달 전374회 조회
1개 답변
0

Hello there,

From the errors seen in the description, the solution could be adding a "timestamp": "UTC" config as per the github issue https://github.com/confluentinc/kafka-connect-storage-cloud/issues/335 which seems relevant.

Also, the other reason could be with the connector configuration which might require tuning.

To confirm the same, please reach back to us over a support case with all the information such as MSK Connector ARN, Configuration, timestamp of when the issue started.

AWS
지원 엔지니어
답변함 3달 전
profile picture
전문가
검토됨 2달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠