By using AWS re:Post, you agree to the Terms of Use

Code running slow on Sagemaker notebook instance for the first time it runs

0

Hello!

I've an issue running the code on SageMaker. I am running my code on SageMaker, which runs my code slowly for the first time, but runs with proper speed, the second time around (I guess there's something getting stored in the cache). Few days back, it was running with the same speed all the time. Whatever I run, be it a model (The model which took just 5 minutes for one epoch when it worked fine estimates 3 hours of running time) / just a code reading the data present in my files, it runs too slow. What could be a possible solution for this? I tried changing the notebook instance types as well, but in vain. I've been struggling for 2 days. It'll be great if someone could help me out a bit soon so that I progress ahead in my project. Thanks in advance!

Edited by: vbsrinivasan on Jun 10, 2020 10:47 AM

asked 2 years ago378 views
7 Answers
0

Can confirm the speed issues. Migrated yesterday to Sagemaker and code runs very slowly the first time, the second time is way faster but is slowing down again mid training. With the same code and training data, training on a K80 is way slower than on Colabs K80.

answered 2 years ago
0

Yeah, you are right. Also, there's one update. Like, whenever I turn off the notebook instance and turn on once again, the code runs very very slowly. But, after running the code once, if I don't turn off the notebook, it runs faster. But, this incurs a lot of cost for me. It would be great if someone from AWS or an expert responds to this!

answered 2 years ago
0

Can you confirm if this is the notebook taking time to spin up the kernel, then load the libraries (at the start of your script presumably) or if this is all cells taking longer to run?

Edited by: MikeChambers on Jun 13, 2020 4:20 AM

answered 2 years ago
0

Hello!

The notebook is taking time to run every cell. Not just libraries and stuff.

answered 2 years ago
0

Exact same experience here. Seeing substantial variation in runtimes between instance restarts. Running the exact same code can take up to a factor of 3 longer (regardless of whether this is just I/O, model training or something else entirely). I had originally attributed this to slow EBS I/O (which by experience has been patchy in the past) but doesn't seem to be related. Real showstopper for sagemaker at this point.

answered 2 years ago
0

Having the same problem too. But for me it is mainly disk I/O. So every time I stop and restart the notebook instance, I need to re-download the data even though they are sitting right there on disk, because if I don't, then it takes a insane amount of time to load the data (even slower than re-download the data and load them). Quite annoying but have not idea how to fix it.

answered 2 years ago
0

Yeah, I've narrowed it down to disk I/O. Extremely slow on first read -- as if the files aren't on the EBS volume but downloaded from elsewhere. Moving away from Sagemaker NBs now for interactive work

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions