Error when saving custom metrics in SageMaker Experiments through SageMaker Pipelines Training Job

0

IHAC that I am working on enabling sagemaker experiments through a training job using SageMaker Pipelines. The below is the logic inserted into the train script which was working fine a few days ago tracking custom metrics into the trial component created by SageMaker Pipelines.

    try:
        print('>>> Loading an existing trial component')
        my_tracker = Tracker.load()
        
    except ValueError:
        print('>>> Creating a new trial component')
        my_tracker = Tracker.create()
        
    my_tracker.log_metric("mse:mse error", mean_squared_error(valid_y, preds))
    
    my_tracker.close()

However, since yesterday I am facing an error with running the same code with the following error:

Loading an existing trial component Traceback (most recent call last): File "training.py", line 82, in <module> my_tracker = Tracker.load() File "/miniconda3/lib/python3.7/site-packages/smexperiments/tracker.py", line 161, in load _ArtifactUploader(tc.trial_component_name, artifact_bucket, artifact_prefix, boto3_session), AttributeError: 'NoneType' object has no attribute 'trial_component_name'

I tried to change the versions of sagemaker and sagemaker-experiments to an older version but still see the same issue. This code works when I trigger just the training job out of SageMaker Pipelines but shows the above error when running through SageMaker Pipelines. Any pointers how to fix this?

AWS
已提问 1 年前459 查看次数
1 回答
0

SageMaker Python SDK is using Boto3 as the backend. You may also want to roll back & pin the Boto3 version.

AWS
已回答 1 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则