By using AWS re:Post, you agree to the Terms of Use
/AWS Step Function Output for container services/

AWS Step Function Output for container services

0

When using Step Functions with a container service like EKS or ECS how does output work for those steps? From the below two linked questions it seems as if input is done by either passing the input to the command override or by passing it as an environment variable. How do these jobs return output to be consumed by the following step?

Input questions:

EDIT

According to this blog there isn't a way to output from ECS or Fargate tasks. So the outputs need to be deterministic based on the previous inputs. Can anyone confirm this and does this go for EKS as well?

https://nuvalence.io/blog/aws-step-function-integration-with-ecs-or-fargate-tasks-data-in-and-out

1 Answers
1
Accepted Answer

Hello,

The answer in that re:Post post has great advice on using .waitForTaskToken for being able to have a more dynamic output from ECS/Fargate and is accurate : https://docs.aws.amazon.com/step-functions/latest/dg/connect-ecs.html

I managed to test using a.waitForTaskToken by following the instructions here: https://docs.aws.amazon.com/step-functions/latest/dg/callback-task-sample-sqs.html

The way the example works (based on the instructions and I confirmed on my end):

  1. Step Functions passes a message that includes a task token to an Amazon Simple Queue Service (Amazon SQS) queue.
  2. Step Functions then pauses, waiting for that token to be returned.
  3. The Amazon SQS queue triggers an AWS Lambda function that calls SendTaskSuccess with that same task token. Below is the code showing how this can be done using a Lambda function:
console.log('Loading function');
const aws = require('aws-sdk');

exports.lambda_handler = (event, context, callback) => {
    const stepfunctions = new aws.StepFunctions();

    for (const record of event.Records) {
        const messageBody = JSON.parse(record.body);
        const taskToken = messageBody.TaskToken;

        const params = {
            output: "\"Callback task completed successfully.\"",
            taskToken: taskToken
        };

        console.log(`Calling Step Functions to complete callback task with params ${JSON.stringify(params)}`);

        stepfunctions.sendTaskSuccess(params, (err, data) => {
            if (err) {
                console.error(err.message);
                callback(err.message);
                return;
            }
            console.log(data);
            callback(null);
        });
    }
};
  1. When the task token is received, the workflow continues.
  2. The "Notify Success" task publishes an Amazon Simple Notification Service (Amazon SNS) message that the callback was received.

So we can apply this to ECS/Fargate now based on the understanding above since Step Function services ECS/Fargate integration support Task Tokens:

  1. Step Functions invokes ECS/Fargate with a task token
  2. Step function then pauses waiting for that token to be returned.
  3. ECS/Fargate sends back a SendTaskScucess with that same task token. You can use the code mentioned above as an example to scaffold a solution from. You can see from the code the following:
        const params = {
            output: "\"Callback task completed successfully.\"",
            taskToken: taskToken
        };

where can modify “output” to have any type of message you’d like. They use \"Callback task completed successfully.\" but you can change this to be whatever you like such as a location of where data is stored in S3/DynamoDB/elsewhere.

  1. When the task token is received the workflow continues.

From the blog post you linked where it says there isn’t a way to output from ECS or Fargate tasks and mention storing in S3, I believe they’re talking about writing data (let’s use a .jpeg image as an example). So following that blog post then, you can write your .jpeg image to S3, and then return in the “output” the location of where the .jpeg image was stored in S3 so that the next step in your Step Function knows the location and can continue to doing some processing by giving the location to another task in the step function.

SUPPORT ENGINEER
answered 2 months ago
  • Thank you for the very thorough answer to this question. This really covers my question and have what I need to continue with this.

    Cheers!

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions