MSK Connect Postgres connector fails during getting Glue avro schema details

Question

I have problem using MSK Postgres Debezium connector with Glue Schema Registry avro serialisation, and getting "connect timed out" to GSR. Logs are following:
```
[Worker-051272e114b69c525] [2022-08-17 08:47:55,387] ERROR [route-events-connector|task-0] WorkerSourceTask{id=route-events-connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:191)
...
com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter.fromConnectData(AWSKafkaAvroConverter.java:97)
[Worker-051272e114b69c525] 	at org.apache.kafka.connect.storage.Converter.fromConnectData(Converter.java:63)
[Worker-051272e114b69c525] 	at org.apache.kafka.connect.runtime.WorkerSourceTask.lambda$convertTransformedRecord$2(WorkerSourceTask.java:313)
[Worker-051272e114b69c525] 	at com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter.fromConnectData(AWSKafkaAvroConverter.java:95)
[Worker-051272e114b69c525] 	... 15 more
[Worker-051272e114b69c525] Caused by: com.amazonaws.services.schemaregistry.exception.AWSSchemaRegistryException: Failed to get schemaVersionId by schema definition for schema name = key-schema
com.amazonaws.services.schemaregistry.common.AWSSchemaRegistryClient.getSchemaVersionIdByDefinition(AWSSchemaRegistryClient.java:144)
[Worker-051272e114b69c525] 	... 28 more
[Worker-051272e114b69c525] Caused by: java.net.SocketTimeoutException: connect timed out
[Worker-051272e114b69c525] 	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
```

Related to GSR connector config:
```
...
key.converter.region=eu-central-1
key.converter.registry.name=my-schema-registry
key.converter.schemaAutoRegistrationEnabled=true
key.converter.schemaName=key-schema
key.converter.avroRecordType=GENERIC_RECORD
key.converter=com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter
value.converter.region=eu-central-1
value.converter.registry.name=my-schema-registry
value.converter.schemaAutoRegistrationEnabled=true
value.converter.schemaName=value-schema
value.converter.avroRecordType=GENERIC_RECORD
value.converter=com.amazonaws.services.schemaregistry.kafkaconnect.AWSKafkaAvroConverter

```
We have already configured MSK connect json Postgres connectors which are working fine and publishing data to MSK topics.

Has anyone successfully configured MSK Connect with Glue Schema Registry for avro serialization? Thanks.

Answer

Hi,

Please note that it is possible to configure MSK Connect with Glue Schema Registry for avro serialization as seen below:

-- https://github.com/awslabs/aws-glue-schema-registry#using-kafka-connect-with-aws-glue-schema-registry

After following the above instructions if you still face issues, I request you to please open a case with AWS Support with the following information:

1.  ARN of MSK
2. ARN of failed connector
3.  Cloudwatch logs of connector
4. Connector configuration

Please open a support case with AWS Support using the following link:

https://support.console.aws.amazon.com/support/home?#/case/create

Answer

In our case VPC endpoint creation for GSR helped whereas all services were within same VPC. Following connector property pointing to that endpoint made connector work properly:
```
value.converter.endpoint=https://vpce-XYZ.glue.our-region.vpce.amazonaws.com
```
hope this will help someone as well.

MSK Connect Postgres connector fails during getting Glue avro schema details

Hi, thank you, submitted case to AWS support.

関連するコンテンツ