Why is IPsec/Phase 2 for AWS Site-to-Site VPN failing to establish a connection?

Accelerating SageMaker Training Jobs running on AWS Trainium

Why is IKE (phase 1 of my VPN tunnel) failing in Amazon VPC?

How do I troubleshoot errors when running Amazon SageMaker training jobs?

How can I test the upload speed of Amazon S3 Transfer Acceleration for a specific file size?

Hmm... have you also checked the training job's metrics in CloudWatch? If so, maybe you could add any interesting findings about that to your question https://docs.aws.amazon.com/sagemaker/latest/dg/training-metrics.html

Hi! 

As of a few days ago, the "**Uploading**" phase of my SageMaker training jobs jumped from **2 minutes to 3+ hours.** The size of my artifacts did not change, but I did enable check-pointing (although this shouldn't affect the zipping and S3 upload of a different directory taking place).

Is there any way to see what SageMaker is doing during that time? I have set `sagemaker_container_log_level = 10` (debug), but no additional logs are published. (and I assume that anything after `Training` will not be logged.

SageMaker "Uploading" phase grew from 2min -> 3h with no changes to artifact size

Developer Tools

Management & Governance

Machine Learning & AI

http.client.RemoteDisconnected at the end of a SageMaker training job

Sagemaker training job successful but model not uploaded to s3

How to checkpoint SageMaker model artifact during a training job?

How to resolve "Artifact upload failed" error in SageMaker?

SageMaker "Uploading" phase grew from 2min -> 3h with no changes to artifact size

Relevant content