AWS Step Function Output for container services

Hello,

The answer in that re:Post post has great advice on using .waitForTaskToken for being able to have a more dynamic output from ECS/Fargate and is accurate : https://docs.aws.amazon.com/step-functions/latest/dg/connect-ecs.html

I managed to test using a.waitForTaskToken by following the instructions here: https://docs.aws.amazon.com/step-functions/latest/dg/callback-task-sample-sqs.html

The way the example works (based on the instructions and I confirmed on my end):

1. Step Functions passes a message that includes a task token to an Amazon Simple Queue Service (Amazon SQS) queue.
2. Step Functions then pauses, waiting for that token to be returned.
3. The Amazon SQS queue triggers an AWS Lambda function that calls SendTaskSuccess with that same task token. Below is the code showing how this can be done using a Lambda function:

console.log('Loading function');
const aws = require('aws-sdk');

exports.lambda_handler = (event, context, callback) => {
    const stepfunctions = new aws.StepFunctions();

    for (const record of event.Records) {
        const messageBody = JSON.parse(record.body);
        const taskToken = messageBody.TaskToken;

        const params = {
            output: "\"Callback task completed successfully.\"",
            taskToken: taskToken
        };

        console.log(`Calling Step Functions to complete callback task with params ${JSON.stringify(params)}`);

        stepfunctions.sendTaskSuccess(params, (err, data) => {
            if (err) {
                console.error(err.message);
                callback(err.message);
                return;
            }
            console.log(data);
            callback(null);
        });
    }
};

4. When the task token is received, the workflow continues.
5. The "Notify Success" task publishes an Amazon Simple Notification Service (Amazon SNS) message that the callback was received.

So we can apply this to ECS/Fargate now based on the understanding above since Step Function services ECS/Fargate integration support Task Tokens:

1. Step Functions invokes ECS/Fargate with a task token
2. Step function then pauses waiting for that token to be returned.
3. ECS/Fargate sends back a SendTaskScucess with that same task token. You can use the code mentioned above as an example to scaffold a solution from. You can see from the code the following:

        const params = {
            output: "\"Callback task completed successfully.\"",
            taskToken: taskToken
        };

where can modify “output” to have any type of message you’d like. They use \"Callback task completed successfully.\" but you can change this to be whatever you like such as a location of where data is stored in S3/DynamoDB/elsewhere.

4. When the task token is received the workflow continues.

From the blog post you linked where it says there isn’t a way to output from ECS or Fargate tasks and mention storing in S3, I believe they’re talking about writing data (let’s use a .jpeg image as an example). So following that blog post then, you can write your .jpeg image to S3, and then return in the “output” the location of where the .jpeg image was stored in S3 so that the next step in your Step Function knows the location and can continue to doing some processing by giving the location to another task in the step function.