How to verify that checkpoints work for SageMaker Spot Training?

0

Hi,

How can we know that checkpoint works before launching a sagemaker spot training job? Is there a way to force a regular checkpoint to s3 instead of waiting for the SIGTERM?

cheers

AWS
전문가
질문됨 4년 전304회 조회
1개 답변
0
수락된 답변

Hi olivier, If you enable Sagemaker checkpointing , it periodically saves a copy of the artifacts into S3. I have used this in pytorch and it works by checkpointing periodically and the blog on Managed Spot Training: Save Up to 90% On Your Amazon SageMaker Training Jobs also mentions the same

To avoid restarting a training job from scratch should it be interrupted, we strongly recommend that you implement checkpointing, a technique that saves the model in training at periodic intervals

AWS
전문가
답변함 4년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠