AWS sagemaker AI autoscale not working as expected

Question

Hello I have deployed a ML model in sagemaker multi model endpoint with instance type ml.m5.xlarge. It takes 6 sec to inference. i am using aws multi model server

I have celery 15 celery worker  which fetches inference request from sqs queue, parallely.
autoscale  has max  15  endpoints.

i see even if 15 endpoints are on but only certain   endpoints are  getting request and rest are not getting any. also the SMIT  end to end  total time is more than 40 sec on an average, even if there are no model loading unloading.

# **this is my sagemaker autoscale configuration.**

```
scalable_dimension:"sagemaker:variant:DesiredInstanceCount"
min_capacity:3
max_capacity:15
policy_name:"SageMaker Endpoint Invocation ScalingPolicy"
policy_type:"TargetTrackingScaling"
TargetValue:2
PredefinedMetricType:"SageMaker Variant Invocations PerInstance"
ScaleInCooldown:600
ScaleOutCooldown:15 
```

# **this is my multi model server configuration **

```vmargs=-XX:+UseContainerSupport -XX:InitialRAMPercentage=8.0 -XX:MaxRAMPercentage=10.0 -XX:-UseLargePages -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError
model_store=/opt/ml/model
load_models=ALL
inference_address=http://0.0.0.0:8080
management_address=http://0.0.0.0:8081
preload_model=true
# management_address=unix:/tmp/management.sock
# number_of_netty_threads=0
# netty_client_threads=0
# default_response_timeout=120
default_workers_per_model=1
# job_queue_size=100
# async_logging=false
# number_of_gpu=1
# cors_allowed_origin
# cors_allowed_methods
# cors_allowed_headers
# keystore=src/test/resources/keystore.p12
# keystore_pass=changeit
# keystore_type=PKCS12
# private_key_file=src/test/resources/key.pem
# certificate_file=src/test/resources/certs.pem
# max_response_size=6553500
# max_request_size=6553500
# decode_input_request=false
# enable_envvars_config=false
```

re:Post Agent · Answer

Your SageMaker auto scaling configuration isn't distributing requests evenly across all available endpoints, which is causing performance issues. There are several potential issues to address:

1. **Request Distribution**: The uneven distribution of requests across your endpoints suggests that some models may be cached in memory while others are experiencing cold starts. This is common in multi-model endpoints where SageMaker removes models from memory when they're not frequently used.

2. **Memory Configuration**: Your JVM settings are limiting memory usage to only 8-10% of available RAM (InitialRAMPercentage=8.0, MaxRAMPercentage=10.0). This is likely too restrictive for a multi-model endpoint and could be causing memory pressure.

3. **Auto Scaling Metric**: You're using "SageMaker Variant Invocations PerInstance" as your scaling metric, but for multi-model endpoints, "MemoryUtilization" might be more appropriate since model loading/unloading is memory-intensive.

4. **Workers Configuration**: You have only 1 worker per model (default_workers_per_model=1), which may be insufficient for your 15 celery workers sending parallel requests.

Consider these improvements:
- Increase the MaxRAMPercentage to allow more models to stay in memory
- Change your scaling metric to MemoryUtilization with a target value around 80%
- Increase default_workers_per_model to handle more concurrent requests
- Consider using a larger instance type with more memory if you're experiencing frequent cold starts
- Check CloudWatch metrics like ModelCacheHit and ModelLoadingWaitTime to better understand your endpoint's performance

If your model is time-sensitive, you might want to consider using real-time inference with a single model instead of a multi-model endpoint to avoid cold start latency.  
**Sources**  
[Troubleshoot SageMaker AI multi-modal endpoint issues | AWS re:Post](https://repost.aws/knowledge-center/sagemaker-multi-modal-endpoint-issues)  
[Automatic scaling of Amazon SageMaker AI models - Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/endpoint-auto-scaling.html)  
[Auto scaling policy overview - Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/endpoint-auto-scaling-policy.html)

AWS sagemaker AI autoscale not working as expected

this is my sagemaker autoscale configuration.

this is my multi model server configuration

Add your answer

Relevant content