Code running slow on Sagemaker notebook instance for the first time it runs

Can confirm the speed issues. Migrated yesterday to Sagemaker and code runs very slowly the first time, the second time is way faster but is slowing down again mid training. With the same code and training data, training on a K80 is way slower than on Colabs K80.

Yeah, you are right. Also, there's one update. Like, whenever I turn off the notebook instance and turn on once again, the code runs very very slowly. But, after running the code once, if I don't turn off the notebook, it runs faster. But, this incurs a lot of cost for me. It would be great if someone from AWS or an expert responds to this!

Can you confirm if this is the notebook taking time to spin up the kernel, then load the libraries (at the start of your script presumably) or if this is all cells taking longer to run?

Edited by: MikeChambers on Jun 13, 2020 4:20 AM

Hello!

The notebook is taking time to run every cell. Not just libraries and stuff.

Exact same experience here. Seeing substantial variation in runtimes between instance restarts. Running the exact same code can take up to a factor of 3 longer (regardless of whether this is just I/O, model training or something else entirely). I had originally attributed this to slow EBS I/O (which by experience has been patchy in the past) but doesn't seem to be related. Real showstopper for sagemaker at this point.

Having the same problem too. But for me it is mainly disk I/O. So every time I stop and restart the notebook instance, I need to re-download the data even though they are sitting right there on disk, because if I don't, then it takes a insane amount of time to load the data (even slower than re-download the data and load them). Quite annoying but have not idea how to fix it.

Yeah, I've narrowed it down to disk I/O. Extremely slow on first read -- as if the files aren't on the EBS volume but downloaded from elsewhere. Moving away from Sagemaker NBs now for interactive work

Sagemaker studio is very slow and buggy. The file explore is very slow it and sometime the kernel does not start. Very disapproving.