StepFunctions/Lambda and ECS

Question

Hello All

Here is what I am attempting to do using StepFunctions/Lambda/ECS+Fargate+python+docker

I have a job that is queuing a request and polling to see if the request is complete. Once complete, the file for the request needs to be downloaded. All the above steps, except downloading file is done using Lambda and Choices in Step functions. For the actual download I am using a container as the download can take over 15 mins.

Question: What I need to do is send the Job_ID from The final "Is File Ready?" Choice to the Fargate/ECS Container so that the python script in the container downloads the file and puts it in S3. The final choice, is basically checking if the file status is Ready or Pending. If Ready, the file is ready to download, else keep looping after 10 mins sleep and check the status again.

Is this the correct thought process: Somehow in the "Is File Ready?" state the output should include the json that the ECSTaskRun can consume? and this json output should have the Job_id as an evn parameter? Once this is done, the python script can read the job_id as env param, os.environ.get('job_id), and do the remaining items?

Or, am I on the wrong track here.

Also I am deploying everything in Python CDK.

Thanks for the feedback.

MichaelFischer · Answer

To pass data from Step Functions to an ECS, you have two options:

1. Override the `command` for the container; or 
2. Override the `environment` for the container (i.e., declare environment variables)

The values can be referenced from the state's input by referring to [paths](https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-paths.html) in it.

Some examples are provided in the [Step Functions integration documentation](https://docs.aws.amazon.com/step-functions/latest/dg/connect-ecs.html).

To pass data from your container back to Step Functions, you need to invoke the ECS Task asynchronously (using `.waitForTaskToken` instead of `.sync`) and pass `$$.Task.Token` to your container, for example, via a `TASK_TOKEN` environment variable. After your container has finished processing, and before your container exits, it must call [SendTaskSuccess](https://docs.aws.amazon.com/step-functions/latest/apireference/API_SendTaskSuccess.html) with the task token and whatever output object you wish to send. This callback pattern is discussed in the [documentation here](https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-wait-token).

StepFunctions/Lambda and ECS

Relevant content