1 Answer
- Newest
- Most votes
- Most comments
3
In order to pass the input parameters to the choice step, you can use a "Pass" state between "Run Glue Crawlers for all LOB" and "Choice" states. This "Pass" state will help you to merge the output of "Run Glue Crawlers for all LOB" and the original input parameters. Update your Step Function definition as follows:
{
"Comment": "A utility state machine to run all Glue Crawlers that match tags",
"StartAt": "Pass",
"States": {
"Pass": {
"Type": "Pass",
"Next": "Get Glue Crawler List based on Client, Environment"
},
"Get Glue Crawler List based on Client, Environment": {
"Next": "Run Glue Crawlers for all LOB",
"OutputPath": "$.CrawlerNames",
"Parameters": {
"Tags.$": "$.tags"
},
"Resource": "arn:aws:states:::aws-sdk:glue:listCrawlers",
"Retry": [
{
"BackoffRate": 5,
"ErrorEquals": [
"States.ALL"
],
"IntervalSeconds": 2,
"MaxAttempts": 3
}
],
"Type": "Task"
},
"Run Glue Crawlers for all LOB": {
"Iterator": {
"StartAt": "Run Glue Crawler for each LOB",
"States": {
"Run Glue Crawler for each LOB": {
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"Next": "Crawler Success"
}
],
"Next": "Crawler Success",
"Parameters": {
"Input": {
"crawler_name.$": "$"
},
"StateMachineArn": "arn:aws:states:ca-central-1:467688788830:stateMachine:run_each_crawler"
},
"Resource": "arn:aws:states:::states:startExecution.sync:2",
"Retry": [
{
"BackoffRate": 5,
"ErrorEquals": [
"States.ALL"
],
"IntervalSeconds": 2,
"MaxAttempts": 1
}
],
"Type": "Task"
},
"Crawler Success": {
"Type": "Succeed"
}
}
},
"ResultPath": null,
"Type": "Map",
"Next": "MergeResults",
"MaxConcurrency": 4
},
"MergeResults": {
"Type": "Pass",
"ResultPath": "$.CrawlerResults",
"Next": "Choice"
},
"Choice": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.glue.environment",
"StringEquals": "qa",
"Next": "start glue-qa-dh"
},
{
"Variable": "$.glue.environment",
"StringEquals": "uat",
"Next": "start glue-uat-dh"
}
]
},
"start glue-qa-dh": {
...
},
"start glue-uat-dh": {
...
}
}
}
This updated definition adds a "MergeResults" Pass state that is used to store the output of the "Run Glue Crawlers for all LOB" state in a new field named "CrawlerResults". Since the "ResultPath" is set to "$.CrawlerResults
Relevant content
- asked a year ago
- Accepted Answerasked a year ago
- asked 3 months ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
- How can I use a Lambda function to automatically start an AWS Glue job when a crawler run completes?AWS OFFICIALUpdated 2 years ago