Ground Truth Job Validation post completion

2

We annotated 10 pdf files in Ground Truth, how do I validate the annotations done by the team ? Do we have any metrics ? Is there a way to get summary of daily task completion details ?

Some example data that I want to get is Number of documents labelled everyday, the count of documents annotated by each person every day.

This metric is very important to track the progress and make a follow up.

Can Ground truth experts provide some insights in this ?

1 回答
0

This is a machine learning problem and goes beyond AWS Ground Truth.

Usually you do not get to measure how confident each annotation is, unless you asked the annotators to say how confident they are on each annotation.

Usually you have to provide some gold standard annotations that you think are correct. You do this on some sample PDFs that you annotated yourself. Then you check the performance of each annotator on your set. You can compute metrics such as Cohen's Kappa to assess agreement between annotators: https://en.wikipedia.org/wiki/Cohen%27s_kappa

已回答 1 年前
  • Sorry for not giving clarity in the question. I actually want the work completion status. Is there a way to get summary of daily task completion details ? Some example data that I want to get is Number of documents labelled everyday, the count of documents annotated by each person every day.

    This metric is very important to track the progress and make a follow up.

    (I made an edit in the question) Thanks.

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则