Sagemaker Pipelines throws ClientError: Missing property value for reference

Question

Hello,

I am trying to use a property from a tuning step into a condition:

```
cond_lte = ConditionLessThanOrEqualTo(
    left=JsonGet(
        step_name=step_tuning.name,
        s3_uri=Join(
            on='/',
            values=['s3:/',default_bucket, step_tuning.properties.BestTrainingJob.TrainingJobName, 'evaluation.json']),
        json_path="regression_metrics.test_rmse.value",
    ),
    right=rmse_threshold,
)
step_cond = ConditionStep(
    name="CheckRMSEEvaluation",
    conditions=[cond_lte],
    if_steps=[step_register_best],
    else_steps=[],
    depends_on=[step_tuning],
)
```
I followed the examples [here](https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_model_building_pipeline.html#tuningstep). However, the pipeline fails in the ConditionStep with this error message:

```
ClientError: Missing property value for reference [Steps.TuningStep.BestTrainingJob.TrainingJobName]
```
I also tried `step_tuning.properties.TrainingJobSummaries[0].TrainingJobName`, same error.
What could be the issue?

re:Post Agent · Answer

The error you're encountering suggests that the pipeline is unable to access the property you're trying to reference from the tuning step. This issue can occur for a few reasons:

1. Timing: The ConditionStep might be trying to access the property before it's available. The tuning step needs to complete before its properties can be accessed.

2. Property reference: The way you're referencing the property might not be correct for your specific use case.

3. Step dependency: Ensure that your ConditionStep explicitly depends on the tuning step.

To address this, try the following:

1. Double-check that your ConditionStep has a dependency on the tuning step. You've included this in your code snippet, which is good.

2. Instead of using the direct property reference, try using a JsonGet function to retrieve the property. This approach is more robust and can handle cases where the property might not be immediately available. For example:

```python
JsonGet(
step_name=step_tuning.name,
property_file=step_tuning.properties.TrainingJobSummary,
json_path="TrainingJobName"
)
```

3. If you're still encountering issues, you might need to use a Lambda step to retrieve the best training job name after the tuning step completes. This ensures that the property is available before the condition is evaluated.

4. Verify that your tuning step is configured correctly and is actually producing the expected outputs.

5. Check the logs of your tuning step to ensure it's completing successfully and generating the expected properties.

If these suggestions don't resolve the issue, you may need to examine the full pipeline definition and the exact configuration of your tuning step to identify any potential misconfigurations or timing issues.  
**Sources**  
[Troubleshooting Amazon SageMaker Pipelines - Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-troubleshooting.html)  
[Workflows for Machine Learning - Amazon SageMaker Pipelines](https://aws.amazon.com/sagemaker/pipelines/)  
[PipelineExecution - Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_PipelineExecution.html)

Sagemaker Pipelines throws ClientError: Missing property value for reference

Relevant content