Sagemaker, Ground Truth labeling works for a while then throws an exception

Question

Trying to build a custom model for comprehend with sagemaker.  I am following these instructions:

https://aws.amazon.com/blogs/machine-learning/custom-document-annotation-for-extracting-named-entities-in-documents-using-amazon-comprehend/

After we annotate a number of pdfs it ends up failing and we are losing our work.  It shows this error:

ClientError: Exception invoking the Lambda function arn:aws:lambda:us-west-2:673752831772:function:annotate-app-GTPreHumanTaskLambdaFunction-foQlysRUOqLC. LambdaErrorCode: Unhandled. Ensure the Lambda function exists, that the role arn:aws:iam::673752831772:role/annotate-app-SageMakerExecutionRole-TUB8KYAWYCUV has permissions to invoke it and try your request again.

401 UnknownError: The specified key output/bonds-job-20230721-labeling-job-20230721T172728/manifests/output/output.manifest isn't present in the S3 bucket comprehend-semi-structured-docs-us-west-2-673752831772.

I am not sure why it's looking for that file and it isn't there.  I can get through 50 pages of annotation and it looks like that lambda function is working properly until it crashes.  Any ideas?

Answer

Figured it out after digging through the lambda logs.  A few of our PDFs had encryption turned on to prevent page extraction and printing.  Bummer it causes the job to fail.

Answer

Hi,

Sometimes the console will say something like "401 ValidationException" but that's not the actual problem. So you should go via CLI or API to obtain a more accurate description of the issue:

please, try:

```
aws sagemaker describe-labeling-job --labeling-job-name yourjobname
```

No guarantee that you'll get a better message but give it a try!

Best.

Didier

Sagemaker, Ground Truth labeling works for a while then throws an exception

Relevant content