Ground Truth Job Validation post completion

2

We annotated 10 pdf files in Ground Truth, how do I validate the annotations done by the team ? Do we have any metrics ? Is there a way to get summary of daily task completion details ?

Some example data that I want to get is Number of documents labelled everyday, the count of documents annotated by each person every day.

This metric is very important to track the progress and make a follow up.

Can Ground truth experts provide some insights in this ?

demandé il y a un an208 vues
1 réponse
0

This is a machine learning problem and goes beyond AWS Ground Truth.

Usually you do not get to measure how confident each annotation is, unless you asked the annotators to say how confident they are on each annotation.

Usually you have to provide some gold standard annotations that you think are correct. You do this on some sample PDFs that you annotated yourself. Then you check the performance of each annotator on your set. You can compute metrics such as Cohen's Kappa to assess agreement between annotators: https://en.wikipedia.org/wiki/Cohen%27s_kappa

répondu il y a un an
  • Sorry for not giving clarity in the question. I actually want the work completion status. Is there a way to get summary of daily task completion details ? Some example data that I want to get is Number of documents labelled everyday, the count of documents annotated by each person every day.

    This metric is very important to track the progress and make a follow up.

    (I made an edit in the question) Thanks.

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions