how to get the "RuleResults" of a Glue Studio DataQuality job

Question

How to get the "RuleResults" generated by a glue studio job that have some dataquality rules, in the dataQuality Tab of the job i can manually download the results to a file when appears the "RuleResults". i have one step function where calls that job, i would like to know the output of that file(where was generated and the key in s3, too) to evaluate in a next step(i.e lambda function) which rules were'nt passed and which ones yes. tkx

Answer

Glue API only returns Job Id in the response. AWS Glue Jobs are designed not to return values.

If the file from Glue is getting uploaded to s3, a separate Lambda can be triggered using S3 event itself. After evaluation, the data can be uploaded to a dynamoDB or another datastore which then can be retrieved by Lambda function configured in state machine with a wait step for s3 triggered Lambda to complete processing.

Helpful documentations:
https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html
https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-wait-state.html
https://meurub.medium.com/receiving-values-from-a-aws-glue-job-ebc6b7386008
https://stackoverflow.com/questions/58222209/how-to-return-output-from-aws-glue-jobs-back-to-the-calling-step-function-workfl

how to get the "RuleResults" of a Glue Studio DataQuality job

관련 콘텐츠