MLFlow on SageMaker - Long-running on-premise experiment tracking

Question

How do you manage token expiry? Is there any way to have long-running training with IAM IDC or do we need to move to long-lived credentials for this use case?

With a long-lived portal session (limit is 90days), will sagemaker-mlflow lib / the SDK automatically refresh the token when the permission-set-based role session expires?

Paul Santus - TerraCloud · Accepted Answer

Answer from the package maintainer:

The sagemaker-mlflow lib is called at every request. Its init method (also invoked on each request) resolves credentials from boto3 at every request. (Upon each mlfow request to the tracking server, it [resolves auth](https://github.com/mlflow/mlflow/blob/3fb7d387c7706e8be3c6d7f5621f98494b51b4aa/mlflow/utils/rest_utils.py#L178), and [resolves the auth registry](https://github.com/mlflow/mlflow/blob/3fb7d387c7706e8be3c6d7f5621f98494b51b4aa/mlflow/tracking/request_auth/registry.py#L53), which ends up [initiating](https://github.com/aws/sagemaker-mlflow/blob/703ed330c73d735003c1bd22d5ca97caafe656ba/sagemaker_mlflow/auth_provider.py#L39) an AuthBoto instance.)

As boto3 manages credentials refresh, when using IAM Identity Center, tracking lasts for as long as the user portal session doesn't expire.

MLFlow on SageMaker - Long-running on-premise experiment tracking

Add your answer

Relevant content