Hello,
There is not a hard limit on the size of a model used on a serverless endpoint as such, but there certain limits that you will reach in practice.
1st a serverless endpoint can have a limited amount of RAM memory (6GB) so even with MMS_DEFAULT_WORKERS_PER_MODEL=1, if your model requires more RAM to make a prediction then you will encounter errors.
2nd is that when you deploy the endpoint it needs to be up+running within 180s (3mins). Part of this time includes loading your model into memory. The larger the model the longer it will take to load into the memory. Unfortunately, I don't have any specific benchmarks on how long this might be against size of model.
3rd you need to consider the cold start issue of your endpoint. This is connected to 2nd point but wanted to mention separately as well. Part of the greatness of serverless endpoints is that when not needed they are completely shut down (cold) and upon request they get spun up and make a prediction. The larger the model, the longer it will take for it be loaded into memory and the longer your cold start time will be.
Relevant questions
Tracking the lineage between Amazon SageMaker endpoint model and Model Monitor captured data
Accepted AnswerHow to configure a serverless sagemaker endpoint?
asked 7 months agoWhat is a practical Inferentia limit to model size?
asked 10 days agowhat is the model(transformer) size limitation in sagemaker serverless endpoint deployment?
asked 7 months agoCloudformation Deployment for Serverless Sagemaker Model Endpoint
asked 3 months agoI want to deploy my model as a Serverless inference
asked a month agoHow to create a serverless endpoint configuration?
Accepted Answerasked 7 months agoSageMaker Model Registry - how to set the Stage column of a Model Package?
asked 10 months agoWhat is the maximum size for a lambda docker image ?
Accepted Answerasked 2 years agoWhy I can't create a serverless sagemaker endpoint anymore ?
Accepted Answerasked 6 months ago
