Hello,
I have tried to train Amazon Rekognition Custom Label object localization model on 105,000 images. I have tried training the model 2 times and reduced training datasets to 70,000 and 55.000, but all I am getting a "training failed" message after almost 48 hours of training with status message: "Amazon Rekognition experienced a service issue."
The model seemed can successfully trained when I used a smaller subset of the data with only about 60 labeled images. Is the dataset size the issue? I found in web, "AWS Rekognition CustomLabels supports upto 250,00 images for Object Detection/Localization." I was also able to train image classification model use Amazon Rekognition Custom Label on 105,000 images successfully with no issue.
AWS Region in which I ran these trainings: us-west-2
AWS account: XXXX-XXXX-5633
Approximate time of trainings: 12/22/2022-12/28/2022, with failed model name list:
- *****.2022-12-27T19.35.49
- *****.2022-12-26T16.04.56
- *****.2022-12-25T16.19.44
- *****.2022-12-22T20.43.45
Any help would be appreciated! Thank you.
I retried training but still failed with "Amazon Rekognition experienced a service issue".