How to automate the checking of Mechanical Turk Workers responses?

We would like to set up an automated system that uses existing training data to check whether workers have correctly understood the task we have for them (it is a data classification task that is more complicated than most). This is so that we can give more work to those workers who perform well and avoid those who perform less well, so that when we move on to new data that needs classifying, we can do so with confidence in the workers involved.

Does anyone have experience of how to do this/is able to help us set this up?