Auto rollback with Guardrails if model accuracy is not good

Question

**While deploying model with guardrails for SageMaker Inference Endpoint::**

We can create CloudWatch alarms to monitor Endpoint performance for metrics like Invocation5XXErrors, ModelLatency and we can rollback the model to previous version if the threshold is matched. [Different metrics available][https://docs.aws.amazon.com/sagemaker/latest/dg/monitoring-cloudwatch.html]

Please let me know is there any way to achieve any one of the following requirement:

1. I want to the use metrics like "accuracy" returned by endpoint and rollback to previous version if the model performance is consistently not good in the real time.
2. Use the metrics like "accuracy" returned by version2 and compare it with version1 or with the ground truth and rollback model if necessary

Answer

You should take a look at SageMaker Model Monitor capabilities and its integration with SageMaker Pipelines to achieve this.

In this method, you will continuously "monitor the model" for model drift and take downstream actions when you detect the drift. The action could be to send a notification(SNS) or re-trigger the Model training.

Some examples for using SageMaker Model Monitor can be found here

In this method, you will continuously "monitor the model" for model drift and take downstream actions when you detect the drift. The action could be to send a notification(SNS) or re-trigger the Model training.

Some examples for using SageMaker Model Monitor can be found [here](https://github.com/aws/amazon-sagemaker-examples/tree/main/sagemaker_model_monitor)

Auto rollback with Guardrails if model accuracy is not good

Contenido relevante