how to increase the storage of host instance

0

The parameters of a big neural network model can be huge. But the largest storage size of a host instance is only 30G, according to https://docs.aws.amazon.com/sagemaker/latest/dg/host-instance-storage.html. Is there a way to increase the storage volume? I have a model (embeddings) that is very close to 30G and caused a no space error when deploying.

Thanks!

bill10
posta 4 anni fa1703 visualizzazioni
2 Risposte
0
Risposta accettata

The disk size is currently not configurable for SageMaker Endpoints with EBS backed volumes. As a workaround, please use instances with ephemeral storage for your SageMaker endpoint.

Example instance types with ephemeral storage:

The full list of Amazon SageMaker instance types can be accessed here: https://aws.amazon.com/sagemaker/pricing/instance-types/

dlragha
con risposta 4 anni fa
0

Thanks! Using a x5d instance solves the issue.

And a quick note: even though I could download the big model to endpoint now, I got timeout error when loading the model in the endpoint. After some trial and error, I solved it by increasing the timeout value and reducing the number of worker by setting the environment variables. To do it, pass this dict

env={"SAGEMAKER_MODEL_SERVER_WORKERS":"1",
"SAGEMAKER_MODEL_SERVER_TIMEOUT":"1800"}

when creating the model object.

bill10
con risposta 4 anni fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande