Tracking record-level failure from Databrew validation

Question

Hi,

This is question with Databrew data profile capability. I am looking for record-level tracking per rule validation status. This will provide while rules has failed on a record from the dataset.

At least, I am looking to separate records after data profiling, valid vs invalid records.

Thanks,

Answer

Hello,
this functionality is currently not supported, I will provide feedback to the team owning the service.

Today, to solve your issue you could use a implement in a project recipe the same logic in the quality check you are interested in and using a conditional transform associate a value (true/False or 1/0) to a new column isCol1Valid .

when you run the processing job and write the output to your target you could easily find all rows that are not satisfying the specific check.

I am aware this is currently a workaround.

hope it helps

Tracking record-level failure from Databrew validation

Relevant questions

Flexmatch validation doesn't fail if you miss a closing bracket on a rule

Trigger databrew job multiple times

Text data cleaning in databrew

Databrew Data Quality Queries

DataBrew Filter

External IDP's identity token validation failure

Can you export Glue DataBrew recipies as code?