How do I automatically start an AWS Glue job when a crawler run completes?

Resolution

To use AWS Glue triggers to start a job when a crawler run completes, use one of the following methods.

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

AWS Glue console

Complete the following steps:

1. Open the AWS Glue console.
2. In the navigation pane, under Data Integration and ETL, choose Triggers.
3. Choose Add trigger.
4. On the Set trigger properties screen, enter the following information:
   Enter your trigger details.
   For Trigger type, choose Job or crawler event.
   Select your conditional logic.
5. For Watched resources, choose Add a watched resource, and then enter the following information:
   For Resource type, choose Crawler.
   For Crawler name, select your crawler.
   For Status, choose Succeeded., and then choose Add.
   Choose Add.
6. Choose Next.
7. For Target resource type, choose Job.
8. Turn on the Trigger on creation option.
9. Choose Create.

AWS CLI

To create a trigger that can start a job when a crawler run completes, run the following create-trigger command:

aws glue create-trigger --name testTrigger --type CONDITIONAL --predicate 'Logical=AND,Conditions=[{LogicalOperator=EQUALS,CrawlerName=testCrawler,CrawlState=SUCCEEDED}]' --actions JobName=testJob --start-on-creation

AWS Glue API

To use the Python boto3 SDK AWS Glue API to configure triggers, run the following command:

import boto3

client = boto3.client("glue")
response = client.create_trigger(
    Name="testTrigger",
    Type="CONDITIONAL",
    Predicate={
        "Logical": "AND",
        "Conditions": [
            {
                "LogicalOperator": "EQUALS",
                "CrawlerName": "testCrawler",
                "CrawlState": "SUCCEEDED",
            },
        ],
    },
    Actions=[
        {"JobName": "testJob"},
    ],
    StartOnCreation=True,
)

With either the AWS CLI option or the API option, create the testTrigger trigger. Then, configure the trigger to start the job testJob after testCrawler successfully runs.

Note: You must use a trigger to start testCrawler. If you manually start the crawler, then the job doesn't get activated by the trigger. In AWS Glue, all jobs or crawlers can start only if a trigger starts the job or crawler. Make sure that all jobs or crawlers in a dependency chain are descendants of the scheduled or on-demand triggers.

Lambda and EventBridge

Create an AWS Lambda function and an Amazon EventBridge rule. When you choose this option, the Lambda function is always on. The Lambda function monitors the crawler regardless of where or when you start the function. You can also update this method to automate other AWS Glue functions. For more information, see How can I use a Lambda function to automatically start an AWS Glue job when a crawler run completes?

AWS Glue workflows

AWS Glue workflows requires you to start the crawler from the Workflows page on the AWS Glue console. For more information, see How do I use AWS Glue workflows to automatically start a job when a crawler run completes?