Specifying TrainingJob Name for SageMaker pipeline TrainingSetp

Question

I'm aware hot to specify the training job name when calling `fit` on an estiamator, but when I create a pipeline I end up with multiple folders on s3 containing different "bits" and I'd really like to get them all in one place.

For `ProcessingStep` I can do this by specifying the destination for any outputs, i.e.

```
 # s3://bucket_name/pipeline_name//step_name/output/output_name
destination=Join(on="/", values=['s3:/', bucket_name, pipeline_name, ExecutionVariables.PIPELINE_EXECUTION_ID, step_name, "output", output_name])
```

But for training jobs it generates a training job name that looks like `pipelines-ek70862duu3h-TrainModel-ajPtiDGLW6/`, the pattern appears to be `pipelines--step_name-/` - I'm not sure what the last token is but I suspect it's random.

Is it possible to define the training job name, i.e. using ExecutionVariables.PIPELINE_EXECUTION_ID so that I can come up with a scheme that's consistent between each step?

Accepted Answer

Hi, 
Thank you for reaching out. 
After my research, unfortunately, SageMaker pipeline TrainingSetp names and various other steps cannot be customized or forced with a specific name. The random prefixes and suffixes added to the pipeline/steps name is to ensure the job names are unique across different pipeline executions. This helps in the management, flexibility and troubleshooting existing and new pipelines.

Since the pipeline names/steps name can't be customized, your defined custom name used in the pipeline process is Tag automatically to your jobs after execution. To view the tags for the pipeline/steps name, you can click on the executed pipeline/ProcessingJob/TrainingJob from the SageMaker console, and scroll down to view all the tags that was linked to the pipeline/step including your custom name.

Thank you

Specifying TrainingJob Name for SageMaker pipeline TrainingSetp

Relevant content