By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

how to get the "RuleResults" of a Glue Studio DataQuality job

0

How to get the "RuleResults" generated by a glue studio job that have some dataquality rules, in the dataQuality Tab of the job i can manually download the results to a file when appears the "RuleResults". i have one step function where calls that job, i would like to know the output of that file(where was generated and the key in s3, too) to evaluate in a next step(i.e lambda function) which rules were'nt passed and which ones yes. tkx

Topics
ServerlessApplication IntegrationComputeAnalytics
Tags
AWS Step FunctionsAWS LambdaAWS Glue
Language
English
Willi5
asked 2 years ago442 views
1 Answer
0

Glue API only returns Job Id in the response. AWS Glue Jobs are designed not to return values.

If the file from Glue is getting uploaded to s3, a separate Lambda can be triggered using S3 event itself. After evaluation, the data can be uploaded to a dynamoDB or another datastore which then can be retrieved by Lambda function configured in state machine with a wait step for s3 triggered Lambda to complete processing.

Helpful documentations: https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-wait-state.html https://meurub.medium.com/receiving-values-from-a-aws-glue-job-ebc6b7386008 https://stackoverflow.com/questions/58222209/how-to-return-output-from-aws-glue-jobs-back-to-the-calling-step-function-workfl

AWS
SUPPORT ENGINEER
Sumit_K-R
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content