By using AWS re:Post, you agree to the Terms of Use
/StepFunctions/Lambda and ECS/

StepFunctions/Lambda and ECS


Hello All

Here is what I am attempting to do using StepFunctions/Lambda/ECS+Fargate+python+docker

I have a job that is queuing a request and polling to see if the request is complete. Once complete, the file for the request needs to be downloaded. All the above steps, except downloading file is done using Lambda and Choices in Step functions. For the actual download I am using a container as the download can take over 15 mins.

Question: What I need to do is send the Job_ID from The final "Is File Ready?" Choice to the Fargate/ECS Container so that the python script in the container downloads the file and puts it in S3. The final choice, is basically checking if the file status is Ready or Pending. If Ready, the file is ready to download, else keep looping after 10 mins sleep and check the status again.

Is this the correct thought process: Somehow in the "Is File Ready?" state the output should include the json that the ECSTaskRun can consume? and this json output should have the Job_id as an evn parameter? Once this is done, the python script can read the job_id as env param, os.environ.get('job_id), and do the remaining items?

Or, am I on the wrong track here.

Also I am deploying everything in Python CDK.

Thanks for the feedback.

1 Answers

To pass data from Step Functions to an ECS, you have two options:

  1. Override the command for the container; or
  2. Override the environment for the container (i.e., declare environment variables)

The values can be referenced from the state's input by referring to paths in it.

Some examples are provided in the Step Functions integration documentation.

To pass data from your container back to Step Functions, you need to invoke the ECS Task asynchronously (using .waitForTaskToken instead of .sync) and pass $$.Task.Token to your container, for example, via a TASK_TOKEN environment variable. After your container has finished processing, and before your container exits, it must call SendTaskSuccess with the task token and whatever output object you wish to send. This callback pattern is discussed in the documentation here.

answered 5 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions