AWS Glue Schema Registry failing even after proper configurations

Question

Hi AWS Community, I have a MSK Connector setup which pick ups messages from my MSK topic "logs" and put it into my RDS database. I am using kafka avro data format to publish messages to my "logs" topic. I have tested my avro format with confluent docker images on my local and it's working as expected but when I am trying to use it with AWS Glue Schema Registry I am getting below error:

Caused by: com.amazonaws.services.schemaregistry.exception.AWSSchemaRegistryException: Failed to auto-register schema. Auto registration of schema is not enabled.
<---- some logs ------>
Caused by: com.amazonaws.services.schemaregistry.exception.AWSSchemaRegistryException: Failed to get schemaVersionId by schema definition for schema name = logs 
<----- some logs ----->
Caused by: software.amazon.awssdk.services.glue.model.EntityNotFoundException: Schema is not found. (Service: Glue, Status Code: 400, Request ID: <my-request-id>)
<---- some logs --->

I am not getting why I am getting this issue even I have setup the below configurations in my java spring boot application:

aws.region=ap-south-1
aws.schema.registry.name=Notification-Logger-Registry
aws.schema.name=logs
spring.kafka.properties.schema.registry.url=https://glue-schema-registry.ap-south-1.amazonaws.com
spring.kafka.producer.properties.schema.registry.url=https://glue-schema-registry.ap-south-1.amazonaws.com
auto.register.schemas=true

May anyone guide me what I am doing wrong here with my configurations? How can I auto register the schema? In AWS console, I can't find any option for auto register schema for the registry.

re:Post Agent · Answer

Based on the error message and your configuration, it appears that there are a few issues to address:

1. Auto-registration of schemas is not enabled: Despite setting `auto.register.schemas=true` in your configuration, the error suggests that auto-registration is not enabled. This setting might not be recognized or applied correctly.

2. The schema "logs" is not found: The error indicates that the schema with the name "logs" doesn't exist in the registry.

To resolve these issues, consider the following steps:

1. Ensure that you're using the correct property names for auto-registration. The AWS Glue Schema Registry typically uses the property `avro.auto.register.schemas` or `glue.schema.auto.register.schemas` instead of just `auto.register.schemas`. Try updating your configuration to use one of these properties.

2. If auto-registration is not working, you may need to manually register your schema first. You can do this through the AWS Glue console, AWS CLI, or programmatically using the AWS SDK.

3. Double-check that your AWS credentials and permissions are correctly set up. Ensure that the IAM role or user associated with your application has the necessary permissions to interact with the AWS Glue Schema Registry.

4. Verify that the registry name "Notification-Logger-Registry" exists in your AWS Glue Schema Registry. If it doesn't, you'll need to create it first.

5. Make sure that the schema name "logs" matches exactly with what you're trying to use in your Kafka producer.

6. If you're using a custom SerDe library, ensure it's properly configured to work with the AWS Glue Schema Registry.

7. Check that your region setting (`ap-south-1`) is correct and matches the region where your Schema Registry is located.

If the issue persists after making these changes, you may need to manually register your schema before using it. You can do this through the AWS Glue console or using the AWS Glue APIs. Once the schema is registered, your producer should be able to use it without encountering the "Schema not found" error.

Remember that auto-registration, when properly configured, should only need to happen once for a new schema. After the initial registration, the schema should be available for subsequent uses.  
**Sources**  
[How the schema registry works - AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/schema-registry-works.html)  
[RegisterSchemaVersion - AWS Glue](https://docs.aws.amazon.com/glue/latest/webapi/API_RegisterSchemaVersion.html)  
[AWS Glue Schema registry - AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html)

AWS Glue Schema Registry failing even after proper configurations

Add your answer

Relevant content