Is there a solid example for creating ground truth in sage maker model monitoring.

1

Conceptually it's clear that Groudtruth depends on business case, and it has to be correlated to predictions in the model. But I am wondering if there is a better specific example anywhere that I can reference for both understanding the concept as well as any code example.

Just to be clear, I am not asking about AWS Ground truth but about the specific step that creates ground truth for model monitoring jobs.

SBob
asked a year ago445 views
1 Answer
1

Hi SBob, I assume you are wanting to create a baseline for a Model Quality Baseline like this: https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality-baseline.html (Please correct if the assumption is wrong)

For creating the baseline dataset typically, you can use the validation dataset. The baseline job calculates metrics for the model and suggests constraints to use to monitor model quality drift. In this case your Ground Truth is already present as apart of the validation set.

But then moving forward during continuous model quality monitoring you will need: 1. predictions made by the deployed model endpoint and 2. the ground truth data. If you enable data capture on the endpoint, prediction data will be captured in S3 for you.

Now for the ground truth data as you say it will depend on the what your model is predicting and what the business use case is. For an example, for a model predicting customer churn the model, the ground truth data could be if the customer actually left (1) the company or not (0). If you could provide more detail on your use case, input data type and prediction I could give a more focused answer but one option to create Ground Truths for your predictions is Amazon Augmented AI: https://aws.amazon.com/augmented-ai/

For more detailed references and code samples see the SageMaker Immersion Day Lab on Model Monitor:

AWS
answered a year ago
  • Thanks for the answer , Manuwai! Very much helpful. Yes I understood merging of ground truth with captured data.

    I have following further questions on the technical implementation of ground truth merge in Sage maker. I see that when model quality monitoring runs , it runs in two parts.

    Ground truth merge job - I understand it's merging ground truth labels with ongoing captured data. But refer to lab docs, specifically the step Ingestion of Ground truth, where we manually (or a separate automated job) create ground truth and associate with eventId. What then , the ground truth job that kicks off separately does in addition to model monitoring job? Is it only further clubbing eventId,ground truth with prediction (probability) value? If so, we could have done it in the ground truth ingestion step itself? Can you or some one please clarify this further?

    Model monitoring job - Is this the job that is calculating perf. metrics? How ground truth is helping it calculate them?

    Thanks

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions