- Newest
- Most votes
- Most comments
It appears that there is a problem with the size of the payload being sent to the endpoint. While SageMaker supports large payloads, the size of the payload that can be sent is still limited.
You can see if adjusting the batch size when sending the payload solves the problem. You should also check the resource limits for your endpoint to ensure they are appropriate for the size of your payload.
If neither of these options works, you can try increasing the instance size or number of instances in your endpoint to accommodate the larger payload.
Finally, you can use SageMaker's remote debugging feature to investigate what's going on during inference. This allows you to debug your inference code while it is running on the instance.
Relevant content
asked a year ago
Lowering the batch size does fix the problem, but I need to send large batches to the endpoint, so it's a moot point. I tired using larger GPU instances, and that doesn't fix the issue. I'm a little confused as to why async inference claims it's able to process up to a gig of data, when the actual instances fail on payloads less than 20 MB. It seems implied that async inference would automatically partition large payloads to work on given instances, but that doesn't seem to be the case?
While SageMaker Async Inference can indeed handle large payloads, it doesn't automatically partition the data for you. Instead of using the Async Inference endpoint directly, you can use SageMaker Processing Jobs to preprocess the data and split it into smaller chunks. Then, you can invoke the Async Inference endpoint on each chunk of data separately.
Maybe this can be a solution for you
https://sagemaker-examples.readthedocs.io/en/latest/sagemaker_processing/scikit_learn_data_processing_and_model_evaluation/scikit_learn_data_processing_and_model_evaluation.html
I'd prefer not to write a process to break up the data, as that would require quite a bit more logic to get it to work correctly with the async endpoint. Is their anyway to increase CPU amount on GPU's in order to be able to hold more data once the data gets ingested onto the instance? Also, when u say increase the number of instances in my endpoint, are you talking about scaling it out? I'm not sure how that would help, as scaling wouldn't break up the payload.