I try to load Mturk hits for our account.
I use boto3 and ListHits function.
After I download ~ 55 days. I start receiving just error System error code: 1019
.
I restart process and it failed on the same date.
What does it mean ? How I can overcome this problem ? Is this is some limitation for this function / API ?
---------------------------
UPD 2022-05-22:
---------------------------
hello
here is my code
Note: I receiving the same problem with aws cli too .
I'm not sure that problem with code because :
- it load's data during 2 days, and then start failing on specific historical dates (I repeat it 2 times and behaviour the same . It load's data from now to middle of the March and then start failing )
- I picked nextToken and got same issue with was cli
- I continued tries and after 1 day it worked some time and downloaded ~ 1 week of data.
- Now it stop working and token expired.
- And for getting this error I need to parse our data again during next week what is annoying.
AWS CLI USAGE:
aws mturk list-hits --max-items=100 --page-size=100 --starting-token=p2:d*******O
An error occurred (ServiceFault) when calling the ListHITs operation (reached max retries: 2): There was an error with this request. (System error code: 1019 1653038262906)
Code
import json
import logging
import datetime
import boto3
from retry import retry
END_DATE = '2022-01-01 00:00:01.000001'
mturk_client = boto3.client('mturk')
paginator = mturk_client.get_paginator('list_hits')
next_token = None
logging.basicConfig(filename=f'list_hits_{datetime.datetime.now()}.log', level=logging.INFO)
log = logging.getLogger()
counter = 0
while True:
cfg = {
'MaxItems': 100,
'PageSize': 100,
}
if next_token:
cfg['StartingToken'] = next_token
def load_page(cfg, counter):
response_iterator = paginator.paginate(
PaginationConfig=cfg
)
for page in response_iterator:
next_token = page.get('NextToken')
counter += 1
with open('./hits/'+str(counter)+'.json', 'w') as out_f:
json.dump(page, out_f, indent=2, default=str)
return counter, next_token
counter, next_token = retry_call(load_page, fargs=[cfg, counter], tries=10, backoff=2, logger=log, max_delay=120)
Hello, thanks for you response.
I added more details , see section
UPD 2022-05-22:
in a main question .