Auto rollback with Guardrails if model accuracy is not good


While deploying model with guardrails for SageMaker Inference Endpoint::

We can create CloudWatch alarms to monitor Endpoint performance for metrics like Invocation5XXErrors, ModelLatency and we can rollback the model to previous version if the threshold is matched. [Different metrics available][]

Please let me know is there any way to achieve any one of the following requirement:

  1. I want to the use metrics like "accuracy" returned by endpoint and rollback to previous version if the model performance is consistently not good in the real time.
  2. Use the metrics like "accuracy" returned by version2 and compare it with version1 or with the ground truth and rollback model if necessary
asked 10 months ago53 views
1 Answer

You should take a look at SageMaker Model Monitor capabilities and its integration with SageMaker Pipelines to achieve this.

In this method, you will continuously "monitor the model" for model drift and take downstream actions when you detect the drift. The action could be to send a notification(SNS) or re-trigger the Model training.

Some examples for using SageMaker Model Monitor can be found here

answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions