I'm invoking the following task in my step function workflow:
https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-stepfunctions-tasks.SageMakerCreateTrainingJob.html
SageMakerCreateTrainingJob starts the task in an asynchronous manner, i.e. the task returns immediately after I request it to start, even if the training job may take hours to finish.
I want to have another task execute after the training job truly and completely finishes. What is an easy way to accomplish this without having to use the two techniques below?
I know there are two clunky ways to do this:
- Using a combination of a Lambda function that calls the SageMaker API to poll for the status of a training job - i.e. to detect when it is finished - and a wait state to force a poll every X number of minutes or so.
- Another possibility is using step function activities, but that would probably preclude me from using Lambdas as a Lambda activity worker would need to keep waiting (doing a thread wait) and constantly polling. I would also need to worry about the 15 minute Lambda time limit.
