SageMaker "Uploading" phase grew from 2min -> 3h with no changes to artifact size

0

Hi!

As of a few days ago, the "Uploading" phase of my SageMaker training jobs jumped from 2 minutes to 3+ hours. The size of my artifacts did not change, but I did enable check-pointing (although this shouldn't affect the zipping and S3 upload of a different directory taking place).

Is there any way to see what SageMaker is doing during that time? I have set sagemaker_container_log_level = 10 (debug), but no additional logs are published. (and I assume that anything after Training will not be logged.

Keine Antworten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen