Why does inference on Amazon SageMaker Studio notebook take longer time than Google Colab to complete?

0
Amazon SageMaker Studio NotebookGoogle Colab
ml.g4dn.xlargeGPU accelerator

Based on the above specifications that was set up for Amazon SageMaker Studio Notebook and Google Colab, it shows that the environments have similarly hardware resources used, particularly Tesla T4 GPU.

However, when running object tracking frame-by-frame inference workload with the same exact .ipynb code on both platforms, the results show almost 4x difference in terms of time taken to complete.

For e.g. a 13mins video: Amazon SageMaker Studio Notebook - took approx. 2 hours Google Colab - took approx. 30mins

Time taken to process individual frames were also logged but did not see anything abnormal.

AWS
ryansee
gefragt vor 6 Monaten162 Aufrufe
1 Antwort
0
Akzeptierte Antwort

If the logic for logging of individual frames is implemented and displayed on the the Studio UI, it would have been the cause of the "degraded" performance on Amazon SageMaker Studio.

As the actual inference is done in the backend, every frame to be logged has to send back to the Studio UI to be displayed. The volume of the logs could have been too much that the Studio UI could not display them out quick enough when they are being transported back from the backend to the frontend, hence the delay. So in actual fact, the inference was actually completed. Do expect the time taken for the inference to be completed to be the same as Google Colab and (if possible) check the output directory for the output generated.

Workaround: to reduce the number of logs printed on Studio UI. Instead of logging it frame-by-frame, do so every 50 or 100 frames.

A ticket has been raised and this bug in the Studio UI should see a fix. Any updates would be posted here.

AWS
ryansee
beantwortet vor 6 Monaten
profile pictureAWS
EXPERTE
überprüft vor 6 Monaten
  • Hi, you should open a ticket via AWS console to see this one fixed: service teams are not supposed to monitor re:Post for bug reports.

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen