passing parameters in step functions

0

Hello,

Within step function, I want to pass parameters received from input to a Glue:ListCrawler state to run related crawlers inside a Glue and after running all crawlers, I also need to pass the input parameters to a choice step to run glue jobs. Because the output of Glue:ListCrawler is the CrawlerName, I can not use OutputPath to include input parameters. Can anyone help me how to receive input parameters within choice state, in this regard?

step function run with this Json format: { "Comment": "Run Glue workflow", "tags": { "client": "test", "environment": "qa"}, "glue":{ "client": "test", "environment": "qa", "partitionDate" : "2023-03-2" }}

Step function definition:

{ "Comment": "A utility state machine to run all Glue Crawlers that match tags", "StartAt": "Pass", "States": { "Pass": { "Type": "Pass", "Next": "Get Glue Crawler List based on Client, Environment" }, "Get Glue Crawler List based on Client, Environment": { "Next": "Run Glue Crawlers for all LOB", "OutputPath": "$.CrawlerNames", "Parameters": { "Tags.$": "$.tags" }, "Resource": "arn:aws:states:::aws-sdk:glue:listCrawlers", "Retry": [ { "BackoffRate": 5, "ErrorEquals": [ "States.ALL" ], "IntervalSeconds": 2, "MaxAttempts": 3 } ], "Type": "Task" }, "Run Glue Crawlers for all LOB": { "Iterator": { "StartAt": "Run Glue Crawler for each LOB", "States": { "Run Glue Crawler for each LOB": { "Catch": [ { "ErrorEquals": [ "States.ALL" ], "Next": "Crawler Success" } ], "Next": "Crawler Success", "Parameters": { "Input": { "crawler_name.$": "$" }, "StateMachineArn": "arn:aws:states:ca-central-1:467688788830:stateMachine:run_each_crawler" }, "Resource": "arn:aws:states:::states:startExecution.sync:2", "Retry": [ { "BackoffRate": 5, "ErrorEquals": [ "States.ALL" ], "IntervalSeconds": 2, "MaxAttempts": 1 } ], "Type": "Task" }, "Crawler Success": { "Type": "Succeed" } } }, "ResultPath": null, "Type": "Map", "Next": "Choice", "MaxConcurrency": 4 }, "Choice": { "Type": "Choice", "Choices": [ { "Variable": "$.environment", "StringEquals": "qa", "Next": "start glue-qa-dh" }, { "Variable": "$.environment", "StringEquals": "uat", "Next": "start glue-uat-dh" },

질문됨 일 년 전1527회 조회
1개 답변
3

In order to pass the input parameters to the choice step, you can use a "Pass" state between "Run Glue Crawlers for all LOB" and "Choice" states. This "Pass" state will help you to merge the output of "Run Glue Crawlers for all LOB" and the original input parameters. Update your Step Function definition as follows:

{
  "Comment": "A utility state machine to run all Glue Crawlers that match tags",
  "StartAt": "Pass",
  "States": {
    "Pass": {
      "Type": "Pass",
      "Next": "Get Glue Crawler List based on Client, Environment"
    },
    "Get Glue Crawler List based on Client, Environment": {
      "Next": "Run Glue Crawlers for all LOB",
      "OutputPath": "$.CrawlerNames",
      "Parameters": {
        "Tags.$": "$.tags"
      },
      "Resource": "arn:aws:states:::aws-sdk:glue:listCrawlers",
      "Retry": [
        {
          "BackoffRate": 5,
          "ErrorEquals": [
            "States.ALL"
          ],
          "IntervalSeconds": 2,
          "MaxAttempts": 3
        }
      ],
      "Type": "Task"
    },
    "Run Glue Crawlers for all LOB": {
      "Iterator": {
        "StartAt": "Run Glue Crawler for each LOB",
        "States": {
          "Run Glue Crawler for each LOB": {
            "Catch": [
              {
                "ErrorEquals": [
                  "States.ALL"
                ],
                "Next": "Crawler Success"
              }
            ],
            "Next": "Crawler Success",
            "Parameters": {
              "Input": {
                "crawler_name.$": "$"
              },
              "StateMachineArn": "arn:aws:states:ca-central-1:467688788830:stateMachine:run_each_crawler"
            },
            "Resource": "arn:aws:states:::states:startExecution.sync:2",
            "Retry": [
              {
                "BackoffRate": 5,
                "ErrorEquals": [
                  "States.ALL"
                ],
                "IntervalSeconds": 2,
                "MaxAttempts": 1
              }
            ],
            "Type": "Task"
          },
          "Crawler Success": {
            "Type": "Succeed"
          }
        }
      },
      "ResultPath": null,
      "Type": "Map",
      "Next": "MergeResults",
      "MaxConcurrency": 4
    },
    "MergeResults": {
      "Type": "Pass",
      "ResultPath": "$.CrawlerResults",
      "Next": "Choice"
    },
    "Choice": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.glue.environment",
          "StringEquals": "qa",
          "Next": "start glue-qa-dh"
        },
        {
          "Variable": "$.glue.environment",
          "StringEquals": "uat",
          "Next": "start glue-uat-dh"
        }
      ]
    },
    "start glue-qa-dh": {
      ...
    },
    "start glue-uat-dh": {
      ...
    }
  }
}

This updated definition adds a "MergeResults" Pass state that is used to store the output of the "Run Glue Crawlers for all LOB" state in a new field named "CrawlerResults". Since the "ResultPath" is set to "$.CrawlerResults

profile picture
전문가
답변함 일 년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠