Trigger databrew job multiple times

Question

Hi, I am using databrew for data quality checks of S3 files. The files arrive continuously. Every time a  files arrives I have an eventbridge job which triggers the databrew job. I have the below questions:

1) is there a way I can trigger the same Databrew profile job multiple times for different files(for ex: let's say we have files xyz1, xyz2, xyz3, etc). they all have same schema and should trigger the same job.

2) If two different (different schemas) files arrive (ex: xyz1, pqr1), can same profile job  be configured for 2 different files? The rules for these 2 files will be different. I dont think same profile job can be used, but just thought to check.

Thanks!!

Answer

For question 1, define different EventBridge patterns and rules for each file convention that invoke the same profile job.  For question 2, you should use different profile jobs that are invoked using similar EventBridge scenario as in question 1.

Trigger databrew job multiple times

Contenuto pertinente