By using AWS re:Post, you agree to the Terms of Use

MSK Connect Postgres connector fails during getting Glue avro schema details

0

I have problem using MSK Postgres Debezium connector with Glue Schema Registry avro serialisation, and getting "connect timed out" to GSR. Logs are following:

[Worker-051272e114b69c525] [2022-08-17 08:47:55,387] ERROR [route-events-connector|task-0] WorkerSourceTask{id=route-events-connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:191)
...
com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter.fromConnectData(AWSKafkaAvroConverter.java:97)
[Worker-051272e114b69c525] 	at org.apache.kafka.connect.storage.Converter.fromConnectData(Converter.java:63)
[Worker-051272e114b69c525] 	at org.apache.kafka.connect.runtime.WorkerSourceTask.lambda$convertTransformedRecord$2(WorkerSourceTask.java:313)
[Worker-051272e114b69c525] 	at com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter.fromConnectData(AWSKafkaAvroConverter.java:95)
[Worker-051272e114b69c525] 	... 15 more
[Worker-051272e114b69c525] Caused by: com.amazonaws.services.schemaregistry.exception.AWSSchemaRegistryException: Failed to get schemaVersionId by schema definition for schema name = key-schema
com.amazonaws.services.schemaregistry.common.AWSSchemaRegistryClient.getSchemaVersionIdByDefinition(AWSSchemaRegistryClient.java:144)
[Worker-051272e114b69c525] 	... 28 more
[Worker-051272e114b69c525] Caused by: java.net.SocketTimeoutException: connect timed out
[Worker-051272e114b69c525] 	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)

Related to GSR connector config:

...
key.converter.region=eu-central-1
key.converter.registry.name=my-schema-registry
key.converter.schemaAutoRegistrationEnabled=true
key.converter.schemaName=key-schema
key.converter.avroRecordType=GENERIC_RECORD
key.converter=com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter
value.converter.region=eu-central-1
value.converter.registry.name=my-schema-registry
value.converter.schemaAutoRegistrationEnabled=true
value.converter.schemaName=value-schema
value.converter.avroRecordType=GENERIC_RECORD
value.converter=com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter

We have already configured MSK connect json Postgres connectors which are working fine and publishing data to MSK topics.

Has anyone successfully configured MSK Connect with Glue Schema Registry for avro serialization? Thanks.

1 Answer
1

Hi,

Please note that it is possible to configure MSK Connect with Glue Schema Registry for avro serialization as seen below:

-- https://github.com/awslabs/aws-glue-schema-registry#using-kafka-connect-with-aws-glue-schema-registry

After following the above instructions if you still face issues, I request you to please open a case with AWS Support with the following information:

  1. ARN of MSK
  2. ARN of failed connector
  3. Cloudwatch logs of connector
  4. Connector configuration

Please open a support case with AWS Support using the following link:

https://support.console.aws.amazon.com/support/home?#/case/create

SUPPORT ENGINEER
answered a month ago
  • Hi, thank you, submitted case to AWS support.

    Valeriy

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions