How to verify that checkpoints work for SageMaker Spot Training?



How can we know that checkpoint works before launching a sagemaker spot training job? Is there a way to force a regular checkpoint to s3 instead of waiting for the SIGTERM?


asked 3 years ago52 views
1 Answer
Accepted Answer

Hi olivier, If you enable Sagemaker checkpointing , it periodically saves a copy of the artifacts into S3. I have used this in pytorch and it works by checkpointing periodically and the blog on Managed Spot Training: Save Up to 90% On Your Amazon SageMaker Training Jobs also mentions the same

To avoid restarting a training job from scratch should it be interrupted, we strongly recommend that you implement checkpointing, a technique that saves the model in training at periodic intervals

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions