mxnet error encountered in Lambda Function

0

I trained and deployed a semantic segmentation network (mlp2.xlarge) using SageMaker. I wanted to use an AWS Lambda function to send an image to this endpoint and get a mask in return however when I use invoke_endpoint it gives an mxnet error in the logs. Funnily when I use the deployed model from a transformer object from inside the SageMaker notebook the mask is returned properly. Here is my Lambda function code:

import json
import boto3

s3r = boto3.resource('s3')

def lambda_handler(event, context):
    # TODO implement
    
    bucket = event["body"]
    key = 'image.jpg'
    local_file_name = '/tmp/'+key
    s3r.Bucket(bucket).download_file(key, local_file_name)

    runtime = boto3.Session().client('sagemaker-runtime')

    with open('/tmp/image.jpg', 'rb') as imfile:
        imbytes = imfile.read()

    # Now we use the SageMaker runtime to invoke our endpoint, sending the review we were given
    response = runtime.invoke_endpoint(
    EndpointName='semseg-2021-12-03-10-05-58-495', 
    ContentType='application/x-image', 
    Body=bytearray(imbytes))                       # The actual image

    # The response is an HTTP response whose body contains the result of our inference
    result = response['Body'].read()
    
    return {
        'statusCode': 200,
        'body': json.dumps(result)
    }

Here are the errors I see in the logs: mxnet.base.MXNetError: [10:26:14] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.4.x.4276.0/AL2_x86_64/generic-flavor/src/3rdparty/dmlc-core/src/recordio.cc:12: Check failed: size < (1 << 29U) RecordIO only accept record less than 2^29 bytes

1 Answer
0

If you are using the built-in Sagemaker algorithm for Semantic Segmentation , the content type must be "image/jpeg" for inference to accept images. For more details, see https://docs.aws.amazon.com/sagemaker/latest/dg/semantic-segmentation.html

AWS
EXPERT
answered 2 years ago
  • Hi Aparna, yes I went through the link before and I tried using invoke_endpoint with ContentType as image/jpeg but it gave me the same error as before. I'll post the logs here. Exception on /invocations [POST] raise MXNetError(py_str(_LIB.MXGetLastError())) Check failed: size < (1 << 29U) RecordIO only accept record less than 2^29 bytes Stack trace returned 10 entries:

  • Thanks. In that case, I would make sure that the content of the image file is valid. You can add a print(len(imbytes) ) to ensure that the file is not empty or something

  • Hi I did what you said. Tried outputting the length of imbytes with the following code and this is the output I get -

    Response "1047212"

    import json import boto3

    s3r = boto3.resource('s3')

    def lambda_handler(event, context): # TODO implement

    bucket = event["body"]
    key = 'image.jpg'
    local_file_name = '/tmp/'+key
    s3r.Bucket(bucket).download_file(key, local_file_name)
    
    runtime = boto3.Session().client('sagemaker-runtime')
    
    with open('/tmp/image.jpg', 'rb') as imfile:
        imbytes = imfile.read()
    
    return str(len(imbytes))
    
  • Hi Aparna, it turns out this was an error related to size. I was sending too big an image - 2000 by 2000 pixels. I decreased the size of the input and it worked fine. But a new problem I have encountered is that I can't decode the response['Body'].read(). When I do response['Body'].read().decode(), it throws an error - Invalid byte at position 2

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions