Sagemaker Pipelines - Using TuningStep.properties as ProcessingStep arguments

Question

How can I use TuningStep's properties fields as arguments for the ProcessStep script? Some of the properties work normally, like `properties.HyperParameterTuningJobName` and` properties.BestTrainingJob.FinalHyperParameterTuningJobObjectiveMetric.Value`. However, other fields don't work and I get the following error when I try to upsert the Pipeline configuration:
```
ClientError: An error occured (ValidationException) when calling the UpdatePipeline operation: Unknown property reference [Steps.Tuning.BestTrainingJob.TunedHyperparameters]
```
I've tried to use the `CreationTime `property too, with the same error.

My code is something like this:
```
proc_run_args = script_processor.run(
	code = "...",
	arguments = [
		"--tuning-job-name", step_tuning.properties.HyperParameterTuningJobName,
		"--train-loss-metric", step_tuning.properties.BestTrainingJob.FinalHyperParameterTuningJobObjectiveMetric.Value.to_string(),
		"--hyperparameters", step_tuning.properties.BestTrainingJob.TunedHyperparameters.to_string()
	],
	inputs = [ ... ],
	outputs = [ ... ],
)

step_process_metrics = ProcessingStep(
	name = "...",
	step_args = proc_run_args
)
```

Answer

Hello,

I understand that you want to use the TuningStep's properties fields as arguments for the ProcessingStep script.

Usually the TuningStep has its own TuningStep definition along the pipeline that occurs after the Processing step and Training step has completed [1]. To make things easier for you, I have provided in resource [2] a sample notebook that uses a pipeline to perform an Amazon SageMaker Processing job. It comes complete with code and explanations. Resource [3] provides further information on defining a pipeline.

I suggest that you open a Support Case with AWS Technical Support [4]. The Technical Support team will be able to help you with this and provide better recommendations for your use case. Please attach the code, screenshots of error, and the steps you have taken when opening a case so that the error can be replicated.

I hope you found this information helpful. As always, should you have any further questions or require any additional information, please reach out.

Resources:

[1] Pipeline Steps - [https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html]()

[2] Amazon SageMaker Processing job sample notebook - [https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker_processing/scikit_learn_data_processing_and_model_evaluation/scikit_learn_data_processing_and_model_evaluation.ipynb]()

[3] Define a Pipeline - [https://docs.aws.amazon.com/sagemaker/latest/dg/define-pipeline.html]()

[4] Open Support Case - [https://support.console.aws.amazon.com/support/home?region=us-east-1#/case/create ]()

Sagemaker Pipelines - Using TuningStep.properties as ProcessingStep arguments

Relevant content