Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

AWS Glue DQ SchemaMatch Failure

0

I am trying to run the SchemaMatch rule for my s3 file. There is a glue table created already for that file, I used the JSON from there as the parent and created a new table, with a dummy s3 location as the refence schema. There are about 800 columns in the table. The rule is failing consistently. I am unable to figure out why.In the output logs, I can see it is unable to fetch the columns from reference table, and that is going as null. Can someone please provide a suggestion. Earlier the same thing was working fine, but now I just cant seem to make it work.

Topics
AnalyticsDatabase
Tags
Extract Transform & Load DataAWS Glue
Language
English

asked 10 months ago182 views

1 Answer
0
Accepted Answer

When troubleshooting a SchemaMatch rule failure in AWS Glue Data Quality, there are several potential causes to investigate:

  1. Schema Comparison Details: The SchemaMatch rule checks if the schema of your primary dataset matches the schema of a reference dataset. This comparison is done column by column, where both the names and types must be identical for a match. The order of columns doesn't matter, but any discrepancy in column names or data types will cause the rule to fail.

  2. Column Type Support: Verify that all your columns use supported data types. SchemaMatch supports these column types: Byte, Decimal, Double, Float, Integer, Long, and Short. If any columns use unsupported types, this could cause failures.

  3. Schema Drift: Since you mentioned it was working before but not now, check if there's been any schema drift between your primary and reference datasets. Even a single column difference will cause the rule to fail.

  4. Rule Syntax: Ensure your rule syntax is correct. The basic syntax should be:

SchemaMatch "reference_dataset_alias" = 1.0

  1. Key Map Issues: Verify that your key map is suitable for the data frames. This is a common issue with AWS Glue Data Quality rules.

  2. Special Characters: Check if any column names contain special characters that might be causing issues.

  3. Large Ruleset: With 800 columns, you might be encountering an overflow error with a large ruleset.

  4. Permissions: Ensure you have sufficient AWS Lake Formation permissions to access both datasets.

To troubleshoot further:

  • Compare the schemas of both datasets carefully, looking for any discrepancies
  • Check the CloudWatch logs for more detailed error messages
  • Try running the rule on a subset of columns to isolate the issue
  • Verify that both datasets are accessible and properly configured in the Glue Data Catalog
  • Consider refreshing the table metadata in the Glue Data Catalog

If the issue persists, you might need to recreate the reference table with an updated schema that exactly matches your primary dataset.
Sources
SchemaMatch - AWS Glue
Troubleshoot rules and rulesets | AWS re:Post
Troubleshooting AWS Glue Data Quality errors - AWS Glue

answered 10 months ago

  • Anjana
    10 months ago

    Given I copied the schema from the Primary dataset itself, there is no difference in the column name or types, I have validated it still. In Error details: it just says Column Counts do not match. But I can see that the column counts are indeed the same.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content