Hello,
We would like to use AWS SageMaker to run our AI models, but the fact that we can't downscale the instances to 0 is very problematic for us as we'll need to duplicate this infrastructure on our various environments (develop, staging, production) and on multiple regions, and this isn't possible cost-wise. Is there a specific reason why this isn't possible, and can we expect this to change soon? What are the solutions that you would suggest to solve this issue, we were thinking of the following:
- Using Kubernetes + Triton (similar to this blog). The main issue being the complexity of the system.
- Using SageMaker Asynchronous Inference. The issue is that we're not sure of the impact on speed, latency, etc. and having the calls asynchronous adds complexity.
Thank you!
Hello Didier,
Thank you for your answer. I have a few questions regarding SageMaker Serverless Inference:
Thank you for your help!