Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

mturk - create_qualification_type: checkbox with multiple possible answers

0

In my stupid test I need the worker to 'check' dog+mouse, and leave 'cat' un-checked.
I can't make it bring a score other than 0!
does anybody having a working example please?
Here is my code:

I have a questions XML with something like this:

        ...
        <AnswerSpecification>
            <SelectionAnswer>
                <MinSelectionCount>0</MinSelectionCount>
                <MaxSelectionCount>4</MaxSelectionCount>
                <StyleSuggestion>checkbox</StyleSuggestion>
                <Selections>
                    <Selection>
                        <SelectionIdentifier>dog</SelectionIdentifier>
                        <Text>Dog</Text>
                    </Selection>
                    <Selection>
                        <SelectionIdentifier>cat</SelectionIdentifier>
                        <Text>Cat</Text>
                    </Selection>
                    <Selection>
                        <SelectionIdentifier>mouse</SelectionIdentifier>
                        <Text>Mouse</Text>
                    </Selection>
           <... close above tags ...>

With a corresponding AnswerKey:

  <Question>
    <QuestionIdentifier>identify_animals</QuestionIdentifier>
    <AnswerOption>
      <SelectionIdentifier>dog</SelectionIdentifier>
      <AnswerScore>1</AnswerScore>
    </AnswerOption>
    <AnswerOption>
      <SelectionIdentifier>cat</SelectionIdentifier>
      <AnswerScore>0</AnswerScore>
    </AnswerOption>
    <AnswerOption>
      <SelectionIdentifier>mouse</SelectionIdentifier>
      <AnswerScore>1</AnswerScore>
    </AnswerOption>
  </Question>

Then come the value mapping:

  <QualificationValueMapping>
    <PercentageMapping>
      <MaximumSummedScore>2</MaximumSummedScore>
    </PercentageMapping>
  </QualificationValueMapping>
Topics
Machine Learning & AI
Tags
Machine Learning & AI
Language
English
asked 5 years ago331 views
1 Answer
0

IM EIN ANI LI MI LI

Thanks to this answer:
https://stackoverflow.com/questions/12468424/how-to-make-an-answer-key-xml-for-a-multi-answer-checkbox-questionspec

I have two things learned:

  1. There is no need to specify scores of zero. This is the default.

  2. Using the following AnswerKey, the worker will get 100 only if marking dog+mouse and leaving 'cat' unchecked. any other combination will get her 0.

<AnswerKey xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2005-10-01/AnswerKey.xsd">
  <Question>
    <QuestionIdentifier>identify_animals</QuestionIdentifier>
    <AnswerOption>
      <SelectionIdentifier>dog</SelectionIdentifier>
      <SelectionIdentifier>mouse</SelectionIdentifier>
      <AnswerScore>2</AnswerScore>
    </AnswerOption>
  </Question>
  <QualificationValueMapping>
    <PercentageMapping>
      <MaximumSummedScore>2</MaximumSummedScore>
    </PercentageMapping>
  </QualificationValueMapping>
</AnswerKey>

I will add as a note that it is quite weird that Amazon's schema server, mechanicalturk.amazonaws.com, is totally unavailable.

answered 5 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content