How to merge Step Function task result with a node in the input?

0

Let's say I have a Step Functions definition with two tasks before some other stuff. The first task is a call to start an asynchronous job and returns a JobId. The second task takes the JobId as input and returns a JobStatus. I want to keep the the JobId and JobStatus together in a node so they can be referenced later in the state machine.

Input to state machine

{
  "somekey": "someval"
}

Using ResultPath "$.jobContext" in task 1, the input to task 2 becomes

{
  "somekey": "someval",
  "jobContext": {
    "JobId": "123"
  }
}

Using InputPath "$.jobContext" and Parameter "$.JobId" in task 2 references the JobId for task 2 correctly. Task 2 returns an object like {"JobStatus":"SUCCEEDED"}. However, I cannot figure out the use of ResultSelector and ResultPath to yield a merged jobContext node like so:

{
  "somekey": "someval",
  "jobContext": {
    "JobId": "123",
    "JobStatus": "SUCCEEDED"
  }
}

If I set ResultPath to "$.jobContext", then I lose the JobId from the input. The task itself is an AWS service call, so I cannot simply add the JobId into the task result. Is there a way to merge an input node with new children from the task result? Or to reference the state input in the ResultSelector?

The workarounds are to expand the parent context with more distinct nodes (drop notion of jobContext parent) or insert an extra layer of definition like jobContext.id.JobId and jobContext.status.JobStatus, but this is not as clean as colocating related properties under one parent node.

AWS
Ryan_B
asked a year ago4206 views
2 Answers
1

ResultSelector can be used to modify the task return value, however, it only allows you to create a different JSON object, which is not what you want. If you can have your Task to return only the status (e.g., "SUCCEEDED") instead of an object, then using $.jobContext.status in the ResultPath will get you what you want. Otherwise, you will have the not so clean result you mentioned.

profile pictureAWS
EXPERT
Uri
answered a year ago
  • Hi Uri, thanks for your reply. Since these tasks are AWS service calls, I do not have control of the task result format. I also cannot find a means of extracting just a value (not a JSON key-val) using the tooling of Step Functions. For now, I am using one of the described workarounds.

0

There are several ways to merge a Step Function task result with a node in the input:

Using the "ResultPath" field in the task definition: You can specify the path to the node in the input where you want to append the task result. For example, if you want to merge the task result with the "jobContext" node, you can set the "ResultPath" field to "$.jobContext". This will append the task result to the existing "jobContext" node in the input.

Using the "ResultSelector" field in the task definition: You can specify a JSONPath expression to select the desired properties from the task result and append them to the existing node in the input. For example, if you want to append the "JobStatus" property from the task result to the "jobContext" node, you can set the "ResultSelector" field to "$.JobStatus".

Using a Lambda Function: You can use a Lambda function to merge the task result with the input node and return the modified input as output. You can use the "Lambda" task type in Step Functions to invoke the Lambda function and pass the input and task result as arguments.

Using a Map state: You can use a Map state to iterate over an array of items and perform some operation on each item, and you can also pass the array of items and some other data as input and output, and use the "ResultPath" and "ResultSelector" to merge the task result with the array of items.

It's worth noting that all the above mentioned methods are only available in AWS Step Functions, so you'll need to use them in state machine definition, not in the task itself.

You can use the "ResultPath" field in the second task to merge the input and the output of the task by specifying the path to the "jobContext" node in the input and appending the new properties from the task output to that node.

For example, you can use the following ResultPath: $.jobContext.JobStatus

This will append the "JobStatus" property from the task output to the existing "jobContext" node in the input, resulting in the desired output: { "somekey": "someval", "jobContext": { "JobId": "123", "JobStatus": "SUCCEEDED" } }

Alternatively, you can use the "ResultSelector" field to select the desired properties from the task output and append them to the existing "jobContext" node in the input.

For example, you can use the following ResultSelector: $.JobStatus

This will select the "JobStatus" property from the task output and append it to the existing "jobContext" node in the input, resulting in the desired output: { "somekey": "someval", "jobContext": { "JobId": "123", "JobStatus": "SUCCEEDED" } }

You can also use the '$' symbol in ResultPath and ResultSelector, it will reference the input state, and you can merge the output with the input in that way.

It's worth noting that the ResultPath and ResultSelector fields are only available in AWS Step Functions, so you'll need to use them in state machine definition, not in the task itself.

answered a year ago
  • Hi Varun, thanks for your reply. These are good suggestions, though I cannot apply them to tasks which wrap AWS service calls. For now, I am using one of the indicated workarounds which is to add another layer of nodes in my jobContext object. Also, I could not find a way to reference input state in the ResultPath or ResultSelector. I believe these mechanisms only work on the Task Result object.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions