Ground Truth - Label Consolidation, NN Models, and Accuracy Levels

Hi naquent,

I'm an engineer at AWS, and I wanted to offer a response to your three questions.

We have a blog post that describes the active learning process and annotation consolidation methodology in a little bit more detail, which might be helpful in understanding the approach - https://aws.amazon.com/blogs/machine-learning/annotate-data-for-less-with-amazon-sagemaker-ground-truth-and-automated-data-labeling/. The exact nature of the annotation consolidation algorithm used in the service is proprietary. However, there is a large body of work that describes this type of method that might give you a better sense for what is involved [1].

I should add that we typically recommend at least three annotators for all scenarios for exactly the reason indicated in the question. That is, when you only have two annotators, it is difficult to determine the preference that should be given to any individual worker in the presence of disagreements.

The neural networks used by Ground Truth are the same as those available elsewhere on the SageMaker platform. When you run an image classification labeling job, you are leveraging a SageMaker image classification model. There are many writeups of these algorithms in various locations, but a useful starting point would be https://docs.aws.amazon.com/sagemaker/latest/dg/algos.html.
You can definitely access and customize these models! As I mentioned above, when you run Ground Truth labeling jobs, machine learning models will be trained over the course of the labeling job. You will find these in your SageMaker console, e.g., the "Training Jobs" console will show you the training jobs that have been run, and the "Models" console will list the model artifacts that have been created. You can use these to run inference, or retrain / fine tune them as desired depending on your circumstances. Please see one of our sample notebooks for a demonstration of how you can use this model for further training, inference, and hosting - https://github.com/awslabs/amazon-sagemaker-examples/blob/master/ground_truth_labeling_jobs/from_unlabeled_data_to_deployed_machine_learning_model_ground_truth_demo_image_classification/from_unlabeled_data_to_deployed_machine_learning_model_ground_truth_demo_image_classification.ipynb.

This accuracy level is set by the service. You can always assess the labeled data after a labeling job in the console. Every label provided by Ground Truth is associated with a "confidence score" which can be helpful in filtering the output of a labeling job. The sample notebook linked above illustrates a couple of workflows in this regard.

Thanks for being a valued AWS customer. Please don't hesitate to reach back out to us.

1. , e.g., A. P. Dawid and A. M. Skene. 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. J. Royal Stat. Soc. Series C 28, 1 (1979), 20–28. http://www.jstor.org/stable/2346806

Thank you, Jonathan! These resources look to be very helpful. I will look through them and follow up if I have any further questions.