how to increase the storage of host instance

0

The parameters of a big neural network model can be huge. But the largest storage size of a host instance is only 30G, according to https://docs.aws.amazon.com/sagemaker/latest/dg/host-instance-storage.html. Is there a way to increase the storage volume? I have a model (embeddings) that is very close to 30G and caused a no space error when deploying.

Thanks!

bill10
asked 4 years ago1627 views
2 Answers
0
Accepted Answer

The disk size is currently not configurable for SageMaker Endpoints with EBS backed volumes. As a workaround, please use instances with ephemeral storage for your SageMaker endpoint.

Example instance types with ephemeral storage:

The full list of Amazon SageMaker instance types can be accessed here: https://aws.amazon.com/sagemaker/pricing/instance-types/

dlragha
answered 4 years ago
0

Thanks! Using a x5d instance solves the issue.

And a quick note: even though I could download the big model to endpoint now, I got timeout error when loading the model in the endpoint. After some trial and error, I solved it by increasing the timeout value and reducing the number of worker by setting the environment variables. To do it, pass this dict

env={"SAGEMAKER_MODEL_SERVER_WORKERS":"1",
"SAGEMAKER_MODEL_SERVER_TIMEOUT":"1800"}

when creating the model object.

bill10
answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions