Questions tagged with Amazon Mechanical Turk

Whenever Turkers submit their assignment, they receive the error message: "Sorry, we couldn't find that page" When opening the web browser console, one Turker said they received this error associated with the POST from pressing submit: "Uncaught SyntaxError: Unexpected token '&' (at submit:26:52)" The HIT generally seems to work properly, even flashing "Loading next HIT" very briefly after submitting, but then posting the above error. Any ideas how to fix this issue? Thanks!

I am trying to create a HIT where the worker presses a key in response to images being displayed. The goal of the HIT is to have many workers perform the same identifications in a random order. For example, worker 1 sees the images in order 1, 2, 3, 4, while worker 2 sees the images in the order 3, 1, 4, 2. Because the response time is essential, the images are displayed one-at-a-time. The worker sees image 1, responds, and then image 1 is replaced with image 2, etc.. I am not clear on what is the best method for doing this and recording the response times. Currently, my plan is to create an text box for each image to be displayed and storing the response time for that image in its corresponding field. So the amount of time to respond to image 1 would be recorded in "TextBox1", the amount of time to respond to image 2 would be recorded in "TextBox2", and so on. Is there a better way to do this? For example, is there a way to retrieve the console log from the worker? That would allow me to write the data to the console and not require additional objects on the page.

Hi! I am doing a longitudinal study and am trying to prevent previous workers from taking the second part of my survey. In order to do this, I have been following instructions on the MTurk blog [https://blog.mturk.com/tutorial-best-practices-for-managing-workers-in-follow-up-surveys-or-longitudinal-studies-4d0732a7319b] but am still unable to add workers to the qualification I created called "completed my survey already" even when I add all 1's in the column next to each worker ID. When I upload the new CVS file with only the worker ID column, the qualification column, and the update Blockstatus, I keep getting the message that I requested 0 qualification scores. Is there any way I can fix this?

I try to load Mturk hits for our account. I use boto3 and ListHits function. After I download ~ 55 days. I start receiving just error `System error code: 1019` . I restart process and it failed on the same date. What does it mean ? How I can overcome this problem ? Is this is some limitation for this function / API ? **---------------------------** **UPD 2022-05-22:** **---------------------------** hello here is my code **Note**: I receiving the same problem with aws cli too . I'm not sure that problem with code because : - it load's data during 2 days, and then start failing on specific historical dates (I repeat it 2 times and behaviour the same . It load's data from now to middle of the March and then start failing ) - I picked **nextToken** and got same issue with was cli - I continued tries and after 1 day it worked some time and downloaded ~ 1 week of data. - Now it stop working and token expired. - And for getting this error I need to parse our data again during next week what is annoying. AWS CLI USAGE: ``` aws mturk list-hits --max-items=100 --page-size=100 --starting-token=p2:d*******O An error occurred (ServiceFault) when calling the ListHITs operation (reached max retries: 2): There was an error with this request. (System error code: 1019 1653038262906) ``` Code ``` import json import logging import datetime import boto3 from retry import retry END_DATE = '2022-01-01 00:00:01.000001' mturk_client = boto3.client('mturk') paginator = mturk_client.get_paginator('list_hits') next_token = None logging.basicConfig(filename=f'list_hits_{datetime.datetime.now()}.log', level=logging.INFO) log = logging.getLogger() counter = 0 while True: cfg = { 'MaxItems': 100, 'PageSize': 100, } if next_token: cfg['StartingToken'] = next_token def load_page(cfg, counter): response_iterator = paginator.paginate( PaginationConfig=cfg ) for page in response_iterator: next_token = page.get('NextToken') counter += 1 with open('./hits/'+str(counter)+'.json', 'w') as out_f: json.dump(page, out_f, indent=2, default=str) return counter, next_token counter, next_token = retry_call(load_page, fargs=[cfg, counter], tries=10, backoff=2, logger=log, max_delay=120) ```

Trying to use javascript libraries like video.js and video_markers.js with jquery to show markers on given time in the video slider of the video. But after adding the library, AMT gives below error. I am not using any select elements in the crowd form myself, looks like this is for the javascript library added , how can I resolve it, can someone please advise?. Thanks Potential errors We found some potential errors with your usage of Crowd HTML Elements. (Element type SELECT): No 'name' attribute specified. This element won't be included in the submitted answer. Below are the external libraries used: <script src="https://unpkg.com/video.js/dist/video.min.js"></script> <script src='https://cdnjs.cloudflare.com/ajax/libs/videojs-markers/0.7.0/videojs-markers.min.js'></script>

Hi, I am trying to publish a task in AMT, but I am getting the error, "you have exceeded your monthly credit limit". I have signed up for AWS billing and my credit card information is valid and up-to-date. Can anyone kindly help me with what to do in this scenario? I have submitted a request to the support team about this, but I need this resolved urgently. Any help will be greatly appreciated.

Hello, our team is working on testing a group of 6 HITs that we have published on the sandbox, but some workers are only able to complete 2 of the HITs in the group before receiving the message mentioned in the title of this post. After receiving this message they get the same message when trying to complete any other HITs that we publish. We also have team members who have been able to complete all 6 HITs without issue. There are not qualifications set on the HIT and the max number of assignments is much higher than the number of assignments completed before this issue occurred. Any help on this situation would be greatly appreciated!

Hi, We currently have a set of HITs uploaded on Mechanical Turk and we realize that the uptake (the number of workers accepting and submitting our tasks) is very low. It has been 3 days since our HITs were uploaded and I am wondering if the low uptake has to do with the complexity of our HIT. My question is - is there as way to fetch the number of times a HIT was accepted and returned? The current API and data structure for a HIT does not include a field with this information - https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/ApiReference_HITDataStructureArticle.html Any pointers greatly appreciated. Best,

I'm creating a task on AMT using the Crowd entity annotation Elements that let reviewers label each word in a text block. I want to ensure that the reviewer selects a label for every word/character in the text before submitting the job. What I am trying is to use `onsubmit` and a validation function in the script section to check whether the length of the labeled entities added together is equal to the total length of the text block but I can't access the output of the crowd entity annotation. This is what I tried but failed: ``` const answers = document.querySelector('crowd-entity-annotation').value.entities ``` I also tried this, which return 'undefined': ``` const answers = document.querySelector('crowd-entity-annotation').entities ``` I appreciate any help.

Is it possible to change worker requirements for a batch after the batch is published? I think I set my approval rate requirement too high. I have gotten some responses, but not enough. I don't want to republish because I don't want the same workers to answer my survey again, but I would like to increase the number or responses by lowering the threshold. Is this possible?

Hi, I am new to Mturk and very confused about the process for purchasing prepaid Hits. I was following the process described in the FAQ of the Amazon Mturk page (https://www.mturk.com/help#enable_aws_billing): ========================================= How do I purchase prepaid HITs on Amazon Mechanical Turk? Follow these steps to purchase prepaid HITs: 1. From your Amazon Mechanical Turk account, go to My Account -> Purchase Prepaid HITs. 2. Enter in the amount you would like to purchase. 3. Select the credit or debit card on file or enter in new credit or debit card information. 4. Confirm your purchase. Note: As a US Requester, you may be prompted to establish a verified Amazon Payments account if you plan to make a purchase above certain amounts. You can create a verified Amazon Payments account at any time here. ========================================= First of all, I am NOT ABLE TO find "Purchase Prepaid HITs" on "My Account" page. So, I tried to establish "a verified Amazon Payments account" as it directs, and I am in the stage when I encounter "We’re verifying your identity now, and we’ll send you an email when the verification is complete. This can take up to 24 hours. You can’t use your account until we’ve verified your identity." But it has been more than two weeks since I saw that message. What is wrong with my whole process? I really do want to purchase prepaid HITs but I am not able to...

I am a first-time requester via MTurk. I have recently created a survey-based HIT that will take 30 minutes to complete. It is limited to US participants aged over 18 years, and the remuneration is $10. I am looking for quality responses, so I followed the guidelines suggested and limited the HIT to workers with a rating over 97% and 10,000 assignments. The task has been open for around 7 hours and I have had 1 response. Is this normal?

I made a custom qualification task using python and when the worker starts "take test" they will get an external link where I put my custom qualification task. After completion, they got a code and they need to put the code in the input form that I provided for them in the qualification submission form. After that in the worker portal, it is showing "Qualification submitted successfully and pending approval from the requester" in the pending qualification section. Now, in the requester account, I should get the worker id under the "workers" tab. But there is no worker id is there. As I do not get any approval request, I made a worker id by myself to check what is the issue. That's what I found out. If anyone has any idea/solution please let me know.

I am trying to use a select form in an MTURK task. The form appears correctly, but the responses used in the form do not appear in the output CSV I get after running a batch of HITS. Here is my dropdown menu: ``` <select name="boxName" required> "<option value="">Select Box Location (1)</option>" "<option value="">Diamond Lakes</option>" "<option value="">Henry Brigham</option>" "<option value="">Municipal Building</option>" "<option value="">Southside</option>" "<option value="">Warren Rd</option>" "<option value="">Warren Rd</option>" "<option value="">Other</option>" </select> How do I get a workers response passed on to the output csv?

Hello, I can't confirm and publish batch on Amazon mTurk. I get this error: "You have exceeded your monthly credit limit". My AWS account and Amazon mTurk are associated and I just added a valid bank card on my AWS account. So I don't understand the problem. Could you help me? Thank you

Hello I've also asked my question in [StackOverflow](https://stackoverflow.com/questions/70840748/mechanical-turk-cannot-see-my-published-batch-in-the-worker-sandbox) I'm trying to publish a new Image Classification batch, but the task is not published. I didn't ask for any qualification for the workers. Yet, I don't see the task in the worker sandbox: https://workersandbox.mturk.com/projects I'm searching by my name: "huji nlp lab" and the title "Visual Analogies". When I observe the task, I see "0% published", which is believe is a problem. How can I solve it? Thanks.

Hi, Desperately seeking some help!!!! I have to pay a bonus of $1 to 450 workers (two different tasks on M-Turk) as a requirement of my survey design for my PhD. To do this task one by one will take many hours, but I was told the following by Amazon; Bonus to Workers on the UI can only be done individually. You can find steps for doing so here: https://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/GivingaBonus.html Bonus to Workers using the API is also individually, however, you can write a custom code to programmatically work through a list of Worker IDs and assignment IDs to bonus multiple Workers at a time. This isn’t something we can assist with. Here is our documentation to bonus Workers individually using API: https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/ApiReference_SendBonusOperation.html Is there anyone who can help me with the second option (assuming it will be quicker). I am not confident with code, but didn't realise paying a bonus to all workers in a hit would be so hard. Any help would be amazing as I'm a little worried how long it will take me to award these bonuses one by one. Thank you!!!

Hi, I get the following error when I try to login to my MTurk requester account with my AWS credentials (as an IAM user): "✖ There is a problem with your session. Please sign in again." The login information is correct, since I am able to login to the aws.console.amazon.com page. I have also used this account with MTurk a year ago and had no issues, so this issue appears to be recent. Any suggestions on what could be going on here? Thank you very much in advance!

I have a set of ten questions in french for which I would like to generate answrs. These answers can be right, worng or contain some right and some false information. I would need to use mechanical turk to generate these answers and label them for if they are right or wrong. The questions are similar to this one : > Pourquoi cette araignée d'eau ne coule-t-elle pas ? Meaning "Why does this water spider not drown." It can be accopmanied by a picture of a water spider floating. A correct answer would be to talk about the hydrophobic limbs of the spider. A wrong answer would be to simply state that the spider is too light to drown or anything else that could even have nothing to do with the prompt. I am wondering what is the right way to use mTurk to provide such good and bad answers ? I would need answers in french. Would that even be possible using mTurk and to ask people to directly label their answer for right or wrong or unsure ?

I have a csv of sentences which I'd like labelled, and have identified GroundTruth labelling jobs as a way to do this. Having spent some time exploring the service, I have some questions: **1) **I can't find a way to display only particular columns to the labellers - e.g. if the dataset has a column of IDs for each sentence, this ideally shouldn't be shown to labellers **2)** There is either single labelling or multi labelling, but I would like a way to have two sets of single-selection labels, where one captures difficulty of assigning the label: Select one for binary classification a) Yes, b) No Select one for difficulty of classification c) Easy, d) Medium, e) Hard Can this be done using custom HTML? Is there a guide to writing this - the template it gives you doesn't seem to render as-is. **3)** There appears to be a maximum of $1.20 payment per task. Is this the case, and why? **4)** Having not used mechanical turk before, are there ways of ensuring people take the work seriously and don't just select random answers? I can see there's an option to have x number of people answer the same question, but is there also a way to put in unambiguous questions to which we already have a 'pre_agreed_label' every nth question, and remove people from the task if they get them wrong? Thanks!

Hello, I was not able to purchase prepaid HITs on the Amazon Mechanical Turk requester site today (Sept 24). Each time that I tried to make a purchase, I got the error message "There has been an error displaying your payment methods, please refresh and try again", and refreshing the page does not work either...I'm wondering whether anyone has encountered the same issue? How can I add payment to my requester account in this case? Thanks.

Hey everyone I'm new to this. I have a need to have a set of tweets annotated. For example: " I love Disney". Q1. "Do you think this tweet is funny?" Q2. "Do you like this tweet?" Q3. "Do you agree with it?" Is it considered a single HIT, or 3 separate HIT? For payment, is it possible to restrict payment to only those who complete all 3 questions ? Thank you so much.

I attempted to log into my mechanical turk account today and I was notified by the page that loaded in my browser, "There is a problem with your Amazon Mechanical Turk account This account has been suspended by the Amazon Mechanical Turk team." No explanation as to why; no link or information given to contact anyone regarding the matter... I checked my email searching for any thing (queried: "Mechanical Turk"). The last thing I have is a"Annual Privacy Notice and Account Updates", dated 05/03/19. Prior to that the last thing I have is a "Weekly Worker Activity Report" dated 09/21/15. Yes, it's been about five years since I've used the account. What happened, when, and why I'd like to know, and please be reinstated. Thank you

I am trying to create an external HIT and I would like my HIT to open a popup browser where the experiment is hosted. When I tried to implement it, the experiment is shown on the Mturk site instead of opening a popup. I'm wondering how should I approach this?

Hello, My name is Murugananth.o, last many years i am working mturk only, this is my full time job, i completed 850000 hits completed, june 23 was my mturk worker account was locked, then that team ask my identity proof to me, i send my pancard scanned copy to fax on august 8 and today itself , but still now not verifying my pancard, sent many mail appeal mturk team also no reply, already wait 55 days , please help me, my eamil id is:rasammal999@gmail.com Regards, Murugananth.o

Good evening, My account was suspended in May. It's funny because at the time I just rolled my eyes and sighed, assuming that this would all be cleared up by the then-coming weekend. I sent maybe 12-15 emails to the aws-cs-mturk-worker-support-form email address. I got a response to each of them giving me a cut and paste response saying that my case was being looked into and they could not give me any further information on the matter. The justification for why they suspended my account...... "We regret to inform you that your Amazon Mechanical Turk account has been suspended. We took this action because; you violated section 3(b) of the MTurk Participation Agreement. Specifically, your account was closed for using robots, scripts, or other automated methods as a substitute for your human intelligence and independent judgment." I wasn't worried about it because I don't use any sort of automation, strange add-ons or scripts. I had faith that Amazon was a legitimate company that wouldn't steal my earnings after I had spent thousands of dollars on their website and worked on Mturk for several years without issue completing over 300K HITs. As such, I foolishly had my earnings set to auto-deposit once a month. Of course it had to happen that my account would be suspended only two days before my earnings would have been withdrawn. This happened to be my best month for earnings on Mturk too, working every day. Amazon stole well over a thousand dollars from me. For weeks I was frustrated but knew that this would all be sorted out because I was an honest and hardworking member. Now I'm lost, frustrated, confused and cheated as I get an email from mturk-account-appeals@amazon.com, saying.... "Hello, Unfortunately, we are not able to take any additional action regarding your Amazon Mechanical Turk Worker account suspension, or offer any additional insight. Please refer to our earlier correspondence for details regarding this decision. Your account will not be re-opened. We regret that we have not been able to address your concerns to your satisfaction. Thank you for your understanding. Sincerely, Account Specialist Amazon Mechanical Turk" Where do I go from here? I've spent months waiting to get my account back. I've replied to the seemingly generic denial email above but they still haven't replied to me. At this point I'm still not convinced that I've had a human look at my account yet. Please help me.

I've gone through the whole process and have successfully created and received results for an image landmarking task using the crowd-keypoint element and template. However, looking at my results, there are things I'd like to restrict to guarantee cleaner data: 1) I'd like to make every keypoint label required (i.e. task cannot be submitted if a label is missing) 2) I'd like to make every keypoint restricted to one instance (right now, task can be submitted with multiple instances of the same label and I only need one very specific point landmarked for each label); ideally a new placement of a point removes the old instance or gives an error message Does anyone know if it is possible and how to do these types of restrictions with the Keypoint Template? Edited by: MaimoOKF on Jul 30, 2019 2:46 PM Edited by: MaimoOKF on Jul 30, 2019 2:47 PM

Hello, I am trying to create a HIT in MTurk using the python API. Let say a single HIT has 3 images. How would I modify the api call mturk.create_hit to replace the URL of the 3 images in my XML code? I believe the call is something like this: ..... response = mturk.create_hit(**task_attributes, Question = questions_xml.replace('${image_url}', ....)) hit_type_id = response\['HIT']\['HITTypeId'] .... So basically, how would I change the 'Question' attribute to replace the 3 image_url? Thanks

I use externalQuestion post my HITs. My webapp is hosted on s3 and served with a domain name and https using route 53, acm and cloudfront: <https://movietitlecamp.com/?taskId=3> I tested the HIT with sandbox and everything works correctly. I can submit the hit successfully, load the next hit and retrieve the answer on sandbox. However, when I push it on production, workers cannot submit hit. After they answer, they got stuck in a page said "Loading next HIT...". I got this err: console.log("The following error (Failed to execute 'postMessage' on 'DOMWindow') is expected, please do not report"); \[ 'https://worker.mturk.com', ].forEach(function(domain) { parent.postMessage('{\"assignmentId\":\"35GMH2SV3HA8EOV182JRDPV5F1QOEL\",\"answer\":{\"taskId\":\"3\",\"media_selection\":\"0\",\"transcription\":\"Kingsman\"}}', domain); }); I'm sure my submit URL is correct. I appended the answer to the submit url as followed: https://www.mturk.com/mturk/externalSubmit/?assignmentId=35GMH2SV3HA8EOV182JRDPV5F1QOEL&taskId=3&media_selection=0&transcription=Kingsman HITTypeId: 3EP32X463BXG5DPRWRT1ISC729P1HC HITId: 3M7OI89LV0G1507PMCM0HHEM2CTC6H Please help. I read most of the forum discussions but I could not figure out the problem! Edited by: trinity19 on Jul 16, 2019 10:31 AM

I would like to design a HIT where a worker answers questions on a page and then moves on to a following page. The contents and questions on the following page will be based on the answers received in the previous page (with some data processing done on the previous answers). Is this currently supported in MTurk? I saw some previous threads say that's not supported in an internal HIT and must be done through an external website, but these were from 5 years ago. I'm interested in whether MTurk currently support it now.

Hello everyone, I'm trying to design an image classification survey on Mturk. The only issue is that I would like to display multiple images in a single HIT (and get responses from each of those images). Is there any resource - HTML template - out there that could help me design such a survey? Is it possible to do such a survey on Mturk? PS: I am also trying to sample the images randomly from a large corpus of images. So, manually creating columns like image_1 and image_2 won't help. Thanks

Hi I was implementing a HIT html that happens to have an accordion in the left panel for instructions and so on for the workers to interact with. Turns out that, every time a work clicks on the accordion the HIT crashes. Sometimes it shows a pop-up warning about leaving the page, others it gives the message: ``` There was a problem submitting your results for this HIT. This HIT is still assigned to you. To try this HIT again, refresh the page. If this problem persists, you can contact the Requester for this HIT by clicking "HIT Details" above and then clicking "Contact This Requester" at the bottom of the pop up. To return this HIT and continue working on other HITs, click the "Return" button on the top or bottom of the right side of the page. ``` Here is an example code: ``` <!DOCTYPE html> <html> <head> <meta name="viewport" content="width=device-width, initial-scale=1"> <style> .accordion { background-color: #eee; color: #444; cursor: pointer; padding: 18px; width: 100%; border: none; text-align: left; outline: none; font-size: 15px; transition: 0.4s; } .active, .accordion:hover { background-color: #ccc; } .panel { padding: 0 18px; display: none; background-color: white; overflow: hidden; } </style> </head> <body> <h2>Accordion</h2> <button class="accordion">Section 1</button> <div class="panel"> <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.</p> </div> <button class="accordion">Section 2</button> <div class="panel"> <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.</p> </div> <button class="accordion">Section 3</button> <div class="panel"> <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.</p> </div> <script> var acc = document.getElementsByClassName("accordion"); var i; for (i = 0; i < acc.length; i++) { acc[i].addEventListener("click", function() { this.classList.toggle("active"); var panel = this.nextElementSibling; if (panel.style.display === "block") { panel.style.display = "none"; } else { panel.style.display = "block"; } }); } </script> <p><!-- MTURK VIA output --><input id="text" name="text" type="text" /></p> <p><input id="submitButton" name="submitButton" type="submit"/></p> </body> </html> ```

Hi, This is my first time testing the semantic segmentation hit and I would like to know if there is any way to customize the colors of the labels in the hit. I'm making this question because let's say I have an object that is predominantly purple. I would like the color of the label to be of the same tone so to be more intuitive for the turkers to use it. I tried to modify the JSON for the label but it just gives me the following error, wich is frustrating: ``` We found some potential errors with your usage of Crowd HTML Elements. (Element type CROWD-SEMANTIC-SEGMENTATION): Error reading attribute value: [ ... ^ Detailed error message: Bad array at line 1 column 6 of the JSON5 data. Still to read: "" (Element type CROWD-SEMANTIC-SEGMENTATION): attributes should NOT have additional properties ``` Thanks, A