Trigger Glue Crawler with S3 bucket event

0

Hi , I would like to trigger Glue Crawler , once the S3 bucket specific prefix files are being added. Is there any design pattern I can follow, seeking guidance from the experts.

I have gone through this blogpost : https://aws.amazon.com/blogs/big-data/run-aws-glue-crawlers-using-amazon-s3-event-notifications/

But I am looking for more simpler solution than extra resource creation. Thanks,

1 Respuesta
0
Respuesta aceptada

Hello.

Since the S3 event trigger cannot directly trigger the Glue crawler, you need to create a configuration using SNS or SQS as per the documentation you checked.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

I think it is also possible to run the Glue crawler using Glue workflow and EventBridge event using methods other than S3 event triggers.
When using EventBridge event, there is no need to create SNS or SQS.
https://docs.aws.amazon.com/glue/latest/dg/workflows_overview.html

profile picture
EXPERTO
respondido hace 9 meses

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas