Can confirm the speed issues. Migrated yesterday to Sagemaker and code runs very slowly the first time, the second time is way faster but is slowing down again mid training. With the same code and training data, training on a K80 is way slower than on Colabs K80.
Yeah, you are right. Also, there's one update. Like, whenever I turn off the notebook instance and turn on once again, the code runs very very slowly. But, after running the code once, if I don't turn off the notebook, it runs faster. But, this incurs a lot of cost for me. It would be great if someone from AWS or an expert responds to this!
Can you confirm if this is the notebook taking time to spin up the kernel, then load the libraries (at the start of your script presumably) or if this is all cells taking longer to run?
Edited by: MikeChambers on Jun 13, 2020 4:20 AM
The notebook is taking time to run every cell. Not just libraries and stuff.
Exact same experience here. Seeing substantial variation in runtimes between instance restarts. Running the exact same code can take up to a factor of 3 longer (regardless of whether this is just I/O, model training or something else entirely). I had originally attributed this to slow EBS I/O (which by experience has been patchy in the past) but doesn't seem to be related. Real showstopper for sagemaker at this point.
Having the same problem too. But for me it is mainly disk I/O. So every time I stop and restart the notebook instance, I need to re-download the data even though they are sitting right there on disk, because if I don't, then it takes a insane amount of time to load the data (even slower than re-download the data and load them). Quite annoying but have not idea how to fix it.
Yeah, I've narrowed it down to disk I/O. Extremely slow on first read -- as if the files aren't on the EBS volume but downloaded from elsewhere. Moving away from Sagemaker NBs now for interactive work
Notebook Instance Types for SageMaker StudioAccepted AnswerEXPERTasked 2 years ago
how to trigger sagemaker pipeline via code change in github?asked 12 days ago
Determining the "right" instance type running Jupyter notebook in Sagemaker when reading/writing a huge parquet file?asked 3 months ago
Code running slow on Sagemaker notebook instance for the first time it runsasked 2 years ago
Has SAS code ever been successfully ran on SageMaker?Accepted Answer
Is there a solution for multi-user Notebook on SageMaker?Accepted Answer
Process hangs when running on batchasked 3 years ago
AWS SageMaker Notebook Instance is not continuing running the cell when I leave my Laptop to execute the cells over a period of time. Can you tell me how can I solve this?asked a month ago
Access secrets from secrets manager into the code the running EC2 dockerasked 5 months ago
Python code stops automatically on Ec2asked 5 months ago