SageMaker GT Streaming Labelling Job Internal Server Error (Job Failed)

Question

I launched a Sagemaker Ground Truth Labeling Job with a vendor for 3D Point Cloud Object Tracking using the API. Yesterday, my job failed with the reason for failure 
```
InternalServerError: We encountered an internal error. Submit a new job.
```

The last three Cloudwatch logs before I saw the failure were

`{
    "labeling-job-name": "scand-trial2-seq10v2stream",
    "event-name": "BATCH_ANNOTATION_STATUS_EVALUATED",
    "event-log-message": "Batch of annotation tasks completed with status FAILED. "
}`

`{
    "labeling-job-name": "scand-trial2-seq10v2stream",
    "event-name": "EXPORTED_LABELED_MANIFEST",
    "event-log-message": "Labeled manifest written to S3 output location."
}`

`{
    "labeling-job-name": "scand-trial2-seq10v2stream",
    "event-name": "IDLE_TIMER_COUNT_INTERRUPTED",
    "event-log-message": "System detected incoming objects. Timer to monitor idleness was reset."
}`

My vendor confirmed that they had annotations saved before this happened, but my output manifest is empty. In addition, I set the expiration time for my job to be 30 days, and this failed at 20 days. What could have caused this error and is there any known method for retrieving the worker annotations that were saved but not submitted?

Answer

I am afraid to inform you that the "Internal Server Error" can occur due to various reasons, In order to deep-dive further and investigate in to the service level logs we would request you to open a support ticket with the AWS premium support with the following information.

-- Account Id

-- Region

-- Failed Job ARNs and cloudwatch logs

Note :Due to security reason, this post is not suitable for sharing customer's resource and account details.

Thank you.

SageMaker GT Streaming Labelling Job Internal Server Error (Job Failed)

관련 콘텐츠