Help us improve the AWS re:Post Knowledge Center by sharing your feedback in a brief survey. Your input can influence how we create and update our content to better support your AWS journey.
How to verify that checkpoints work for SageMaker Spot Training?
Hi,
How can we know that checkpoint works before launching a sagemaker spot training job? Is there a way to force a regular checkpoint to s3 instead of waiting for the SIGTERM?
cheers
- Tags
- Amazon SageMaker
- Language
- English
- Newest
- Most votes
- Most comments
Hi olivier, If you enable Sagemaker checkpointing , it periodically saves a copy of the artifacts into S3. I have used this in pytorch and it works by checkpointing periodically and the blog on Managed Spot Training: Save Up to 90% On Your Amazon SageMaker Training Jobs also mentions the same
To avoid restarting a training job from scratch should it be interrupted, we strongly recommend that you implement checkpointing, a technique that saves the model in training at periodic intervals
Relevant content
- asked 3 years ago
