what are some ways/alternative to expose sagemaker endpoints as a HTTP /REST endpoints?

Question

I am testing out serverless sagemaker endpoints and  was  planning  to integrate it with api gateway directly, but realized there is a 29 seconds timeout limit in api gateway, which might not work if the endpoints take longer than that for inference. is there any workaround for this apart from adding a lambda in-between? I am trying to avoid lambda as it might add to the latency

Answer

Here is a blog post about adding a API Gateway in front of a SageMaker endpoint: https://aws.amazon.com/blogs/machine-learning/creating-a-machine-learning-powered-rest-api-with-amazon-api-gateway-mapping-templates-and-amazon-sagemaker/
How long does your model take for inference right now? if your model is slower than you expected, you might want to choose a larger instance type or use a GPU instance for models that can use GPU. Take a look at https://github.com/aws-samples/aws-marketplace-machine-learning/blob/master/right_size_your_sagemaker_endpoints/Right-sizing%20your%20Amazon%20SageMaker%20Endpoints.ipynb

You can also consider using async inference. When the API Gateway receives a request, trigger a async inference job and return immediately. Then let the endpoint write the result to a S3 bucket, then notify your user either by SNS -> Email or through a polling API etc.

what are some ways/alternative to expose sagemaker endpoints as a HTTP /REST endpoints?

Relevant content