SageMaker metrics persistence

Question

Quick questions on ML metrics persistence from sagemaker training tasks. The SageMaker regexp-over-CloudWatch is an attractive option, yet the metric retention in Cloudwatch seems to be restricted to 15 days.

1. How to persist those metrics longer? Is it common to extract them out of Cloudwatch regularly to persist them somewhere else, eg S3 or an RDS? what is the best practice for long-term persistence of those
    metrics?
 2. Would SageMaker Experiments allow a collection of similar data (customer-defined training metrics) over a longer retention?

Accepted Answer

1. You can now persist algorithm metrics from SageMaker Training Jobs (the ones you can collect with regexes or the ones available from built-in algorithms by default) by setting [EnableSageMakerTimeSeriesMetrics](https://alpha-docs-aws.amazon.com/sagemaker/latest/dg/API_AlgorithmSpecification.html#SageMaker-Type-AlgorithmSpecification-EnableSageMakerMetricsTimeSeries) through the AWS SDK or `enable_sagemaker_metrics=true` in the SageMaker Python SDK. These metrics are persisted long term, and available through Amazon SageMaker Studio. (Go to "Metrics" -> "Add Chart" from the detail page of a training job). These are available at no additional cost.

2. Yes, SageMaker Experiments allow collection of similar data

Note that system metrics (CPU/GPU/Memory/Disk) are still available only through CloudWatch.

SageMaker metrics persistence

Relevant content