By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Amazon Rekognition experienced a service issue - Why is this error happening?

0

Hello,

I have tried to train an Amazon Rekognition (custom label) object detection model on a set of labeled medical images to detect medical tools. I have about 23,000 labeled images with 8 classes. I have tried training the model 2 times, but both times I am getting a "training failed" message after almost 24 hours of training: "Amazon Rekognition experienced a service issue." How can I solve this? Is there any reason this error is occurring?

The model seemed to successfully train when I used a smaller subset of the data with only about 1500 labeled images. Is the dataset size the issue?

Any help would be appreciated! Thank you.

Topics
Machine Learning & AI
Tags
Machine Learning & AIAmazon RekognitionAmazon SageMaker Ground Truth
Language
English
AWS-User-0102979
asked 2 years ago381 views
4 Answers
1

Hello AWS-User-0102979, thanks for using Amazon Rekognition Custom Labels. Our team has noticed these training jobs that failed due to service issues in the last few days, and is currently investigating what is the root cause. AWS Rekognition CustomLabels supports upto 250,00 images for Object Detection/Localization. Please refer to here: https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/limits.html . Please consider opening a case via AWS Support Center. As son as we identify the problem, we will get back to you. Thanks for reaching out.

Thanks, aws-yasarka

aws-yasarka
answered 2 years ago
0

Hello AWS-User-0102979, Can you please share the AWS Region in which you ran these trainings and approximate time of trainings to help us debug the failures.

aws-yasarka
answered 2 years ago
  • AWS-User-0102979
    2 years ago

    I am not exactly sure what AWS region these trainings were done, but it may have been the US East (N. Virginia) based on my location.

    the last failed trainings were done on:

    2022-02-07 T21.16.02 February 07, 2022 2022-02-06 T09.51.06 February 06, 2022 2022-02-04 T18.29.17 February 04, 2022

0

Please confirm if you region is US West 2 (PDX/Oregon). You can sign-in to you AWS Account Web Console, and you can see the region info on the top right corner. If you are using aws-cli, you can get region info in your AWS config file. We have been informed with multiple training failures between 02/04/2022 and 02/06/2022. We will let you know when we identify and resolve the issue as soon as possible.

AWS
aws-yasarka
answered 2 years ago
0

Hello, Can you please try training again ?

AWS
aws-yasarka
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content