Minimum number of annotators AWS Grounth Truth

When using Amazon SageMaker Ground Truth with Mechanical Turk as your workforce, you can adjust the number of workers that annotate each data object. However, it's important to note that while you can specify a desired number of annotators, there isn't a guaranteed way to force exactly three annotations for every data point.

The number of annotations per object can vary due to several factors, including worker availability, task complexity, and time constraints. Ground Truth uses annotation consolidation to improve the quality of labels, but this process doesn't necessarily ensure a fixed number of annotations for each item.

To increase the likelihood of getting three annotations per data point, you could try the following approaches:

1. Increase the number of workers requested per task beyond three. This may help ensure that at least three workers complete the task, even if some don't finish.

2. Extend the task timeout and expiration times to give more opportunity for workers to complete the annotations.

3. Adjust the price per task to make it more attractive to workers, potentially increasing participation.

4. Monitor the labeling job progress and manually add additional tasks for data points that don't receive the desired number of annotations.

If having exactly three annotations per data point is crucial for your use case, you might need to implement a custom post-processing step. This could involve reviewing the output manifest file, identifying data points with fewer than three annotations, and potentially re-submitting those for additional annotation rounds.

Remember that while more annotators can potentially improve label quality for complex tasks, it may not always make a significant difference for simpler jobs. Consider the complexity of your task and the quality requirements when determining the optimal number of annotators.
Sources
Using the Amazon Mechanical Turk Workforce - Amazon SageMaker AI
Accelerate custom labeling workflows in Amazon SageMaker Ground Truth without using AWS Lambda | AWS Machine Learning Blog