How to log Sagemaker metrics beyond the 40 metrics constraint?

Question

**How can we log metrics for a training job beyond the 40 metrics limit?**

Tried to create a training job for an NER (Named Entity Recognition) task. 
As I have a lot of classes (e.g. PER, ORG, LOC, etc.), I would like to be able to log the associated metrics accordingly.
However, when I run the `estimator.fit()`, it threw the following error.

```
ClientError: An error occurred (ValidationException) when calling the CreateTrainingJob operation: 1 validation error detected: Value '[MetricDefinition(...), ...]' at 'algorithmSpecification.metricDefinitions' failed to satisfy constraint: Member must have length less than or equal to 40
```

Accepted Answer

This is consistent with the [CreateTrainingJob API doc](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AlgorithmSpecification.html#sagemaker-Type-AlgorithmSpecification-MetricDefinitions), and to my knowledge it's a hard (non-adjustable) limit. However if you have a strong requirement, it may be worth raising a support case to double-check whether an increase is possible?

You could consider logging additional metrics to CloudWatch directly from your training script via the CloudWatch [APIs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutMetricData.html) / [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudwatch.html) if needed? I expect there'd be some limitations in where the metrics are visible (e.g. showing on the training job details page in SageMaker console? showing in the Experiments & Trials view in SageMaker Studio?) - but if you were able to get them logged under the same `/aws/sagemaker/TrainingJobs/{TrainingJobName}` namespace as the auto-collected metrics, they might reflect. Your script code should be able to determine the current training job name from the [TRAINING_JOB_NAME environment variable](https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo-running-container.html#your-algorithms-training-algo-running-container-environment-variables) if wanting to try this.

Be aware that (while fast), metric data API calls can take some time: In an ideal world you might do them asynchronously to avoid slowing down your training job.

How to log Sagemaker metrics beyond the 40 metrics constraint?

Relevanter Inhalt