S3 Dataset versioning with SageMaker?


Is there any standard for ML S3 dataset tracking or versioning? Basically, what setup allows to track a given model training execution to a given dataset? Interested to hear about proven or state-of-the-art ideas

asked 5 years ago1703 views
3 Answers

Nowadays, there are 3rd party tool that can be used alongside SageMaker. One example is Data Version Control (DVC), and we have discussed it how to integrate within SageMaker Processing jobs and SageMaker Training Jobs in this blogpost. As an alternative, you can leverage SageMaker Pipelines when your data preparation step is executed as a processing step within a pipeline execution. Pipelines allows you to achieve data versioning in a programmatic way by using execution-specific variables like ExecutionVariables.PIPELINE_EXECUTION_ID, which is the unique ID of a pipeline run. We can, for example, create a unique key for storing the output datasets in S3 that ties them to a specific pipeline run. We have also discussed this possibility as part of this blogpost.

answered 2 years ago
Accepted Answer

Unfortunately, managing versions of datasets and which models used them is not embedded in SageMaker. But, you can use SageMaker search to manage the differences in data location between experiments. In that case, if your dataset isn't too big, my recommendation will be to create a standard for data structure in S3. i.e. for each new dataset, create a new prefix in S3 with your logic. Using SageMaker search you'll be able to find all your jobs and compare between datasets.

answered 5 years ago
profile picture
reviewed 17 days ago

As Paolo_DF suggested, DVC has become a best practice for controlling your dataset, model, and scripts versions. While I utilize Sagemaker Studio to train and deploy my custom ML model, I have found Paolo's blogpost to be challenging to follow. Although I respect their expertise, I prefer not to use the Training job and Sagemaker Experiment services. Instead, I would suggest referring to the following resources: https://dvc.org/doc/user-guide/integrations/sagemaker https://medium.com/analytics-vidhya/versioning-data-and-models-in-ml-projects-using-dvc-and-aws-s3-286e664a7209

answered 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions