I launched a Sagemaker Ground Truth Labeling Job with a vendor for 3D Point Cloud Object Tracking using the API. Yesterday, my job failed with the reason for failure
InternalServerError: We encountered an internal error. Submit a new job.
The last three Cloudwatch logs before I saw the failure were
{ "labeling-job-name": "scand-trial2-seq10v2stream", "event-name": "BATCH_ANNOTATION_STATUS_EVALUATED", "event-log-message": "Batch of annotation tasks completed with status FAILED. " }
{ "labeling-job-name": "scand-trial2-seq10v2stream", "event-name": "EXPORTED_LABELED_MANIFEST", "event-log-message": "Labeled manifest written to S3 output location." }
{ "labeling-job-name": "scand-trial2-seq10v2stream", "event-name": "IDLE_TIMER_COUNT_INTERRUPTED", "event-log-message": "System detected incoming objects. Timer to monitor idleness was reset." }
My vendor confirmed that they had annotations saved before this happened, but my output manifest is empty. In addition, I set the expiration time for my job to be 30 days, and this failed at 20 days. What could have caused this error and is there any known method for retrieving the worker annotations that were saved but not submitted?