Serialization error kafka connect

0

Getting this error when deploying a Kafka connector

Converting byte[] to Kafka Connect data failed due to serialization error of topic

Any ideas what could be causing this and how to resolve it.

  • Problem: Reading non-JSON data with JsonConverter

    If you have non-JSON data on your source topic but try to read it with the JsonConverter, you can expect to see:

    org.apache.kafka.connect.errors.DataException: Converting byte[] to Kafka Connect data failed due to serialization error: … org.apache.kafka.common.errors.SerializationException: java.io.CharConversionException: Invalid UTF-32 character 0x1cfa7e2 (above 0x0010ffff) at char #1, byte #7) This could be caused by the source topic being serialized in Avro or another format.

  • Solution: If the data is actually in Avro, then change your Kafka Connect sink connector to use:

    "value.converter":"io.confluent.connect.avro.AvroConverter", "value.converter.schema.registry.url":"http://schema-registry:8081", OR if the topic is populated by Kafka Connect, and you can and would rather do so, switch the upstream source to emit JSON data:

    "value.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false",

posta un anno fa2641 visualizzazioni
1 Risposta
0

Hi,

The error suggests serialization error with Kafka topic. This can happen when the Kafka Connect is unable to deserialize data from Kafka topic. This is generally caused by mismatch in serialization/deserialization format - data may have been serialized using a different format than what the Kafka Connect expects.

For example, if the data is serialized using Avro, but Kafka Connect configuration is set up for JSON deserialization, it would result in a serialization error.

To resolve this issue, ensure that your Kafka Connect configuration matches the data serialization format. Update the key.converter[1] and value.converter[2] properties in the Kafka Connect configuration to use the correct deserializer.

[1] https://kafka.apache.org/documentation/#connectconfigs_key.converter

[2] https://kafka.apache.org/documentation/#connectconfigs_value.converter

AWS
TECNICO DI SUPPORTO
con risposta un anno fa
AWS
ESPERTO
verificato un anno fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande