Is there a solid example for creating ground truth in sage maker model monitoring.

Hi SBob, I assume you are wanting to create a baseline for a Model Quality Baseline like this: https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality-baseline.html (Please correct if the assumption is wrong)

For creating the baseline dataset typically, you can use the validation dataset. The baseline job calculates metrics for the model and suggests constraints to use to monitor model quality drift. In this case your Ground Truth is already present as apart of the validation set.

But then moving forward during continuous model quality monitoring you will need: 1. predictions made by the deployed model endpoint and 2. the ground truth data. If you enable data capture on the endpoint, prediction data will be captured in S3 for you.

Now for the ground truth data as you say it will depend on the what your model is predicting and what the business use case is. For an example, for a model predicting customer churn the model, the ground truth data could be if the customer actually left (1) the company or not (0). If you could provide more detail on your use case, input data type and prediction I could give a more focused answer but one option to create Ground Truths for your predictions is Amazon Augmented AI: https://aws.amazon.com/augmented-ai/

For more detailed references and code samples see the SageMaker Immersion Day Lab on Model Monitor:

• Instructions: https://catalog.us-east-1.prod.workshops.aws/workshops/63069e26-921c-4ce1-9cc7-dd882ff62575/en-US/lab4/monitoring
• Jupyter Notebook: https://github.com/aws-samples/amazon-sagemaker-immersion-day/blob/master/SageMaker-ModelMonitoring-ModelQuality.ipynb