SageMaker "Uploading" phase grew from 2min -> 3h with no changes to artifact size

0

Hi!

As of a few days ago, the "Uploading" phase of my SageMaker training jobs jumped from 2 minutes to 3+ hours. The size of my artifacts did not change, but I did enable check-pointing (although this shouldn't affect the zipping and S3 upload of a different directory taking place).

Is there any way to see what SageMaker is doing during that time? I have set sagemaker_container_log_level = 10 (debug), but no additional logs are published. (and I assume that anything after Training will not be logged.

No hay respuestas

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas