Trigger Glue Crawler with S3 bucket event

0

Hi , I would like to trigger Glue Crawler , once the S3 bucket specific prefix files are being added. Is there any design pattern I can follow, seeking guidance from the experts.

I have gone through this blogpost : https://aws.amazon.com/blogs/big-data/run-aws-glue-crawlers-using-amazon-s3-event-notifications/

But I am looking for more simpler solution than extra resource creation. Thanks,

1 Answer
0
Accepted Answer

Hello.

Since the S3 event trigger cannot directly trigger the Glue crawler, you need to create a configuration using SNS or SQS as per the documentation you checked.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

I think it is also possible to run the Glue crawler using Glue workflow and EventBridge event using methods other than S3 event triggers.
When using EventBridge event, there is no need to create SNS or SQS.
https://docs.aws.amazon.com/glue/latest/dg/workflows_overview.html

profile picture
EXPERT
answered 8 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions