Code running slow on Sagemaker notebook instance for the first time it runs

Question

Hello!   
  
I've an issue running the code on SageMaker. I am running my code on SageMaker, which runs my code slowly for the first time, but runs with proper speed, the second time around (I guess there's something getting stored in the cache). Few days back, it was running with the same speed all the time. Whatever I run, be it a model (The model which took just 5 minutes for one epoch when it worked fine estimates 3 hours of running time) / just a code reading the data present in my files, it runs too slow. What could be a possible solution for this? I tried changing the notebook instance types as well, but in vain. I've been struggling for 2 days. It'll be great if someone could help me out a bit soon so that I progress ahead in my project. Thanks in advance!  
  
Edited by: vbsrinivasan on Jun 10, 2020 10:47 AM

Answer

Exact same experience here. Seeing substantial variation in runtimes between instance restarts. Running the exact same code can take up to a factor of 3 longer (regardless of whether this is just I/O, model training or something else entirely). I had originally attributed this to slow EBS I/O (which by experience has been patchy in the past) but doesn't seem to be related. Real showstopper for sagemaker at this point.

Answer

Yeah, I've narrowed it down to disk I/O. Extremely slow on first read -- as if the files aren't on the EBS volume but downloaded from elsewhere. Moving away from Sagemaker NBs now for interactive work

Answer

Hello!   
  
The notebook is taking time to run every cell. Not just libraries and stuff.

Answer

Having the same problem too. But for me it is mainly disk I/O. So every time I stop and restart the notebook instance, I need to re-download the data even though they are sitting right there on disk, because if I don't, then it takes a insane amount of time to load the data (even slower than re-download the data and load them). Quite annoying but have not idea how to fix it.

Answer

Can confirm the speed issues. Migrated yesterday to Sagemaker and code runs very slowly the first time, the second time is way faster but is slowing down again mid training. With the same code and training data, training on a K80 is way slower than on Colabs K80.

Answer

Can you confirm if this is the notebook taking time to spin up the kernel, then load the libraries (at the start of your script presumably) or if this is all cells taking longer to run?  
  
Edited by: MikeChambers on Jun 13, 2020 4:20 AM

Answer

Yeah, you are right. Also, there's one update. Like, whenever I turn off the notebook instance and turn on once again, the code runs very very slowly. But, after running the code once, if I don't turn off the notebook, it runs faster. But, this incurs a lot of cost for me. It would be great if someone from AWS or an expert responds to this!

Answer

Sagemaker studio  is very slow and buggy. The file explore is very slow it and sometime the kernel does not start. Very disapproving.

Code running slow on Sagemaker notebook instance for the first time it runs

Relevant content