Sagemaker Notebook keeps hanging/freezing

0

I have been using Sagemaker Studio Notebook and suddenly it started hanging. When this happens, the notebook freezes completely. Than I have to wait some seconds (the delay duration is not constant and is common to reach about 30 seconds) and then it just freezes again, making its usage impossible. I was using a temporary account provided by Udacity and after trying different approaches to find and solve the problem, I switched to a personal account but the problem persists. Approaches I have tried so far:

  • Shutdow and start kernel
  • Restart kernel
  • Restart kernel and clear outputs
  • Log out and Login (from Sagemaker)
  • Log out and Login (from AWS)
  • Change region
  • Trying a different browser (I tried Chrome and Firefox)
  • Trying using other account (personal)

I also checked CloudWatch logs but didn't find anything that seemed unusual.

Thiago
已提問 2 年前檢視次數 314 次
1 個回答
1
已接受的答案

The most likely cause of this from my experience is a (very) large number of active git changes.

Given your "current" working folder (the one you're navigated to in the folder sidebar menu), the jupyterlab-git integration regularly checks if you're inside a git repository and polls for changes in that repository if so.

When this list is very large, I've sometimes seen it cause significant slowdowns in the overall UI because of the way the underlying (open-source) extension works. This has been discussed before for example in this GitHub issue - which is now marked closed but I've still seen it happening.

For example, maybe you (like me 😅) forgot to gitignore a data folder or node_modules and generated thousands of untracked files there: You might see a significant slowdown whenever you're navigated to a folder within the scope of that git repo.

Suggested solution would be:

  • Use the folder sidebar to navigate anywhere other than the affected git repository (e.g. to your root folder?), and you should see the slowdown resolve pretty much immediately if this is the underlying cause
  • Now the tricky task of finding and clearing up the problemmatic folder(s) without navigating to them in the folder GUI:
    • You could use a System Terminal, cd to the affected folder and run git status to see where the many changes are hiding, if you're not sure already
    • Add a .gitignore file (or modify your existing one) to make git ignore those changes. Because it starts with a dot, .gitignore is hidden by default in the JupyterLab file browser anyway. I usually use a system terminal to e.g. cp myrepo/.gitignore gitignore.txt to create a visible copy (somewhere other than the repository folder which you're trying to avoid navigating to!) and then mv gitignore.txt myrepo/.gitignore to overwrite with my edited version

Alternatively (if e.g. it's a folder full of new files that you no longer care about like node_modules) you could just slog through the slowness to delete the problemmatic folder in the UI - but of course the problem would return if you re-created them later without .gitignore.

AWS
專家
Alex_T
已回答 2 年前
  • Thanks a lot Alex! That was the cause! Simply deleting the folder which had thousand of images made the UI responsive again. And then I also noticed that before the deletion the git status bar was permanently in "Refreshing" status. After the deletion, it's quickly toggling between "Refreshing" and "Idle"

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南