跳至內容

What channel name should I use for template.json in SageMaker's estimator.fit()?

0

Referring to SageMaker's estimator.fit():

estimator.fit(inputs=None, wait=True, logs='All', job_name=None, experiment_config=None)

Based on https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-foundation-models-use-python-sdk-estimator-class.html, the Python command is:

estimator.fit(
    {"train": training_dataset_s3_path, "validation": validation_dataset_s3_path}
)

How do I modify this, if I need to include a template.json file? Will this depend on the training container image used? Note that I am using both a training and validation dataset.

AWS

已提問 2 年前檢視次數 133 次

2 個答案
1

When including a template.json file in your SageMaker estimator.fit() method, you should use the channel name "config" for the template.json file. The channel names "train" and "validation" are typically used for training and validation datasets, respectively.

To modify the estimator.fit() call to include the template.json file, you would add another key-value pair to the inputs dictionary. Here's how you can modify the command:

estimator.fit(
    {
        "train": training_dataset_s3_path,
        "validation": validation_dataset_s3_path,
        "config": "s3://your-bucket/path/to/template.json"
    }
)

The choice of channel name for the template.json file may depend on the specific training container image being used. Some containers might expect configuration files in a channel named "config", while others might use a different name. It's important to consult the documentation or specifications of the particular container you're using to ensure you're using the correct channel name.

In general, the "config" channel is a common convention for configuration files like template.json, but always verify this with the specific requirements of your chosen algorithm or container. If you're using a custom container, you would need to ensure that your container is set up to read from the channel name you specify.
Sources
SageMaker Training and Inference Toolkits - Amazon SageMaker AI
create_training_job - Boto3 1.35.76 documentation
create_hyper_parameter_tuning_job - Boto3 1.35.76 documentation

已回答 2 年前

專家

已審閱 2 年前

1

Assuming you're referring to the prompt template JSON in instruction-based fine-tuning as mentioned here:

The training folder can also include a template JSON file (template.json) that describes the input and output formats of your data. If no template file is provided, the following template file is used...

...So I believe what you'd want is for training_dataset_s3_path to point at an S3 prefix containing both your JSON-Lines training data, and a single template.json file. I.e set training_dataset_s3_path = "s3://doc-example-bucket/my-folder"

AWS
專家

已回答 2 年前

專家

已審閱 2 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。