- Newest
- Most votes
- Most comments
Hi,
The error suggests serialization error with Kafka topic. This can happen when the Kafka Connect is unable to deserialize data from Kafka topic. This is generally caused by mismatch in serialization/deserialization format - data may have been serialized using a different format than what the Kafka Connect expects.
For example, if the data is serialized using Avro, but Kafka Connect configuration is set up for JSON deserialization, it would result in a serialization error.
To resolve this issue, ensure that your Kafka Connect configuration matches the data serialization format. Update the key.converter[1] and value.converter[2] properties in the Kafka Connect configuration to use the correct deserializer.
[1] https://kafka.apache.org/documentation/#connectconfigs_key.converter
[2] https://kafka.apache.org/documentation/#connectconfigs_value.converter
Relevant content
- asked 3 months ago
- asked 10 months ago
AWS OFFICIALUpdated 5 months ago- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
Refer to blog https://www.confluent.io/blog/kafka-connect-deep-dive-converters-serialization-explained
Problem: Reading non-JSON data with JsonConverter
If you have non-JSON data on your source topic but try to read it with the JsonConverter, you can expect to see:
org.apache.kafka.connect.errors.DataException: Converting byte[] to Kafka Connect data failed due to serialization error: … org.apache.kafka.common.errors.SerializationException: java.io.CharConversionException: Invalid UTF-32 character 0x1cfa7e2 (above 0x0010ffff) at char #1, byte #7) This could be caused by the source topic being serialized in Avro or another format.
Solution: If the data is actually in Avro, then change your Kafka Connect sink connector to use:
"value.converter":"io.confluent.connect.avro.AvroConverter", "value.converter.schema.registry.url":"http://schema-registry:8081", OR if the topic is populated by Kafka Connect, and you can and would rather do so, switch the upstream source to emit JSON data:
"value.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false",