Why does inference on Amazon SageMaker Studio notebook take longer time than Google Colab to complete?

0
Amazon SageMaker Studio NotebookGoogle Colab
ml.g4dn.xlargeGPU accelerator

Based on the above specifications that was set up for Amazon SageMaker Studio Notebook and Google Colab, it shows that the environments have similarly hardware resources used, particularly Tesla T4 GPU.

However, when running object tracking frame-by-frame inference workload with the same exact .ipynb code on both platforms, the results show almost 4x difference in terms of time taken to complete.

For e.g. a 13mins video: Amazon SageMaker Studio Notebook - took approx. 2 hours Google Colab - took approx. 30mins

Time taken to process individual frames were also logged but did not see anything abnormal.

AWS
ryansee
demandé il y a 6 mois162 vues
1 réponse
0
Réponse acceptée

If the logic for logging of individual frames is implemented and displayed on the the Studio UI, it would have been the cause of the "degraded" performance on Amazon SageMaker Studio.

As the actual inference is done in the backend, every frame to be logged has to send back to the Studio UI to be displayed. The volume of the logs could have been too much that the Studio UI could not display them out quick enough when they are being transported back from the backend to the frontend, hence the delay. So in actual fact, the inference was actually completed. Do expect the time taken for the inference to be completed to be the same as Google Colab and (if possible) check the output directory for the output generated.

Workaround: to reduce the number of logs printed on Studio UI. Instead of logging it frame-by-frame, do so every 50 or 100 frames.

A ticket has been raised and this bug in the Studio UI should see a fix. Any updates would be posted here.

AWS
ryansee
répondu il y a 6 mois
profile pictureAWS
EXPERT
vérifié il y a 6 mois
  • Hi, you should open a ticket via AWS console to see this one fixed: service teams are not supposed to monitor re:Post for bug reports.

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions