using boto3 how can i set up an eventbridge to start a glue job when a crawler completes

Question

I saw the sample implementation for scheduled event calling lambda function at https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/python/example_code/lambda/scheduled_lambda.py

I need to invoke lambda function when a crawler completes using boto3.

thanks,
Ravi.

Justin Callison · Answer

In this case, you might be better off to use a Step Functions workflow. That will allow you to orchestrate running the crawler, then running the job (or whatever else you need to) after.

This blog post might help get you started: https://aws.amazon.com/blogs/compute/orchestrating-aws-glue-crawlers-using-aws-step-functions/

Sample code to go with it here: https://github.com/aws-samples/aws-stepfunctions-examples/tree/main/sam/demo-manage-glue-crawlers-with-step-functions

And if you want to run this whole thing on a schedule, you can do that using the Events property on the `AWS::Serverless::StateMachine` resource.

https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-statemachine.html#sam-statemachine-events

using boto3 how can i set up an eventbridge to start a glue job when a crawler completes

Relevant content