Empty payload in the Pass step of Step Function Step Machine when triggered by an EventBridge event

Question

In a Step Function State Machine, I want to make a decision based on a value passed by an event from EventBridge. I use a Pass step to wait for the event to be triggered, then pass it to a Choice step and make a decision based on the value of a property in the payload of the triggering event.

Upon inspecting the 'input' value of the 'Pass' step, I noticed that it's an empty object. This is unexpected, as I've set the 'Matched event' as the value of the triggering Rule in EventBridge. Despite my attempts to transform the event using dot notation ($.detail), the 'Pass' step still receives an empty object as its input value.

The end goal is that, in the Step Function, I want to wait for a Glue Crawler to finish its job. Then, based on the value of "status" in the event's payload, I decide whether to finish successfully or notify the Data Engineering team.

Can anyone guess what I am doing wrong?
Do you have any ideas about what to check or how to debug?

## Step Function definition:
```json
{
  "StartAt": "start-parallel-glue-jobs",
  "States": {
    "start-parallel-glue-jobs": {
      "Type": "Parallel",
      "Next": "start-glue-crawler",
      "Branches": [
        {
          "StartAt": "start-glue-job-something-as-example",
          "States": {
            "start-glue-something-as-example": {
              "End": true,
              "Type": "Task",
              "Resource": "arn:aws:states:::glue:startJobRun.sync",
              "Parameters": {
                "JobName": "something",
                "Arguments": {
                  "--something": "something"
                }
              }
            }
          }
        }
      ]
    },
    "start-glue-crawler": {
      "Next": "wait-for-crawler-completion",
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:glue:startCrawler",
      "Parameters": {
        "Name": "something-crawler"
      }
    },
    "wait-for-crawler-completion": {
      "Type": "Pass",
      "ResultPath": null,
      "InputPath": "$",
      "Next": "check-crawler-state"
    },
    "check-crawler-state": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.status",
          "StringEquals": "SUCCEEDED",
          "Next": "start-transform-glue-job"
        }
      ],
      "Default": "crawler-failed"
    },
    "crawler-failed": {
      "Type": "Fail",
      "Error": "Crawler failed",
      "Cause": "Crawler failed"
    },
    "start-transform-glue-job": {
      "End": true,
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "some-job-name"
      }
    }
  },
  "TimeoutSeconds": 1800,
  "Comment": "An example"
}
```

## Event Bridge:
Event pattern:
```json
{
  "detail-type": ["Glue Crawler State Change"],
  "source": ["aws.glue"],
  "detail": {
    "crawlerName": ["some-crawler"],
    "state": ["SUCCEEDED", "FAILED", "STOPPED"]
  }
}
```

Input to target: "Matched event"

When I execute the Sfn State machine, the value I see in the "Input" of the `wait-for-crawler-completion` is `{}`
I use `$` for input path.

Pooyan Razian · Accepted Answer

I realized that “Pass” does not wait for the event & it directly passes the empty object to the next step.
Also, Crawler does not support TaskToken in the "wait for token" integration pattern (unlike Lambda integration)

So, I had to use this approach:
https://aws.amazon.com/blogs/compute/orchestrating-aws-glue-crawlers-using-aws-step-functions/

Giovanni Lauria · Answer

The issue you're facing is due to the way AWS Step Functions handles input from EventBridge events. By default, Step Functions expects the event payload to be wrapped in a `detail` object. However, in your case, the event payload is not wrapped, which is why you're seeing an empty object in the `Pass` state.

To resolve this issue, you can use the `InputPath` parameter in the `Pass` state to access the correct event payload. Here's how you can modify the `wait-for-crawler-completion` state:

```json
"wait-for-crawler-completion": {
  "Type": "Pass",
  "ResultPath": null,
  "InputPath": "$.detail",
  "Next": "check-crawler-state"
}
```

By setting `InputPath` to `$.detail`, you're instructing Step Functions to look for the event payload within the `detail` object, which is how EventBridge events are structured.

Additionally, you might need to adjust the `check-crawler-state` state to access the correct property from the event payload. For example, if the `status` property is nested within the `detail` object, you would need to use the following expression:

```json
"check-crawler-state": {
  "Type": "Choice",
  "Choices": [
    {
      "Variable": "$.detail.state",
      "StringEquals": "SUCCEEDED",
      "Next": "start-transform-glue-job"
    }
  ],
  "Default": "crawler-failed"
}
```

In this example, `$.detail.state` is used to access the `state` property from the event payload, assuming it's nested within the `detail` object.

Empty payload in the Pass step of Step Function Step Machine when triggered by an EventBridge event

Step Function definition:

Event Bridge:

Add your answer

Relevant content