How to pass environment variables in sagemaker tuner job

0

Sagemaker training jobs support setting environment variables on-the-fly in the training job:

 "Environment": { 
      "string" : "string" 
   },

https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html

I did not find an equivalent for the tuner jobs:

https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateHyperParameterTuningJob.html

According to my testing, the SagemakerTuner in the python SDK simply ignores the environment variables set in the passed estimator.

Is there any way to pass environment variables to the training jobs started by a tuner job programmatically, or is that currently unsupported?

Chris
asked a year ago1498 views
1 Answer
1
Accepted Answer

Thanks for raising this. Yes, as you point out the Environment collection is not supported in the underlying CreateHyperparameterTuningJob API and therefore the SageMaker Python SDK can't make use of it when running a tuner.

As discussed on the SM Py SDK GitHub issue here, you might consider using hyperparameters instead to pass parameters through to the job?

If you specifically need environment variables for some other process/library, you could also explore setting the variables from your Python script (perhaps to map from hyperparam to env var?).

Or another option could be to customize your container image to bake in the variable via the ENV command? For example to customize an existing AWS Deep Learning Container (framework container), you could:

  • Use sagemaker.image_uris.retrieve(...) to find the base image URI for your given framework, version, region, etc. You'll need to authenticate Docker to this registry as well as your own Amazon ECR account.
  • Create a Dockerfile that takes this base image URI as an arg and builds FROM it, something like this example
  • Add the required ENV commands to bake in the (static) environment variables you need
  • docker build your custom container (passing in the base image URI as a --build-arg), upload it to Amazon ECR, and use in your SageMaker training job.
AWS
EXPERT
Alex_T
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions