Waiting for SageMaker CreateTrainingJob to Finish

0

I'm invoking the following task in my step function workflow:

https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-stepfunctions-tasks.SageMakerCreateTrainingJob.html

SageMakerCreateTrainingJob starts the task in an asynchronous manner, i.e. the task returns immediately after I request it to start, even if the training job may take hours to finish.

I want to have another task execute after the training job truly and completely finishes. What is an easy way to accomplish this without having to use the two techniques below?

I know there are two clunky ways to do this:

  • Using a combination of a Lambda function that calls the SageMaker API to poll for the status of a training job - i.e. to detect when it is finished - and a wait state to force a poll every X number of minutes or so.
  • Another possibility is using step function activities, but that would probably preclude me from using Lambdas as a Lambda activity worker would need to keep waiting (doing a thread wait) and constantly polling. I would also need to worry about the 15 minute Lambda time limit.
asked 3 years ago720 views
1 Answer
0

Oops, I guess I didn't look at the integrationPattern property. Anyway, I set it to RUN_JOB and now the call is synchronous.

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions