Sagemaker Notebook keeps hanging/freezing

0

I have been using Sagemaker Studio Notebook and suddenly it started hanging. When this happens, the notebook freezes completely. Than I have to wait some seconds (the delay duration is not constant and is common to reach about 30 seconds) and then it just freezes again, making its usage impossible. I was using a temporary account provided by Udacity and after trying different approaches to find and solve the problem, I switched to a personal account but the problem persists. Approaches I have tried so far:

  • Shutdow and start kernel
  • Restart kernel
  • Restart kernel and clear outputs
  • Log out and Login (from Sagemaker)
  • Log out and Login (from AWS)
  • Change region
  • Trying a different browser (I tried Chrome and Firefox)
  • Trying using other account (personal)

I also checked CloudWatch logs but didn't find anything that seemed unusual.

Thiago
asked 2 years ago308 views
1 Answer
1
Accepted Answer

The most likely cause of this from my experience is a (very) large number of active git changes.

Given your "current" working folder (the one you're navigated to in the folder sidebar menu), the jupyterlab-git integration regularly checks if you're inside a git repository and polls for changes in that repository if so.

When this list is very large, I've sometimes seen it cause significant slowdowns in the overall UI because of the way the underlying (open-source) extension works. This has been discussed before for example in this GitHub issue - which is now marked closed but I've still seen it happening.

For example, maybe you (like me 😅) forgot to gitignore a data folder or node_modules and generated thousands of untracked files there: You might see a significant slowdown whenever you're navigated to a folder within the scope of that git repo.

Suggested solution would be:

  • Use the folder sidebar to navigate anywhere other than the affected git repository (e.g. to your root folder?), and you should see the slowdown resolve pretty much immediately if this is the underlying cause
  • Now the tricky task of finding and clearing up the problemmatic folder(s) without navigating to them in the folder GUI:
    • You could use a System Terminal, cd to the affected folder and run git status to see where the many changes are hiding, if you're not sure already
    • Add a .gitignore file (or modify your existing one) to make git ignore those changes. Because it starts with a dot, .gitignore is hidden by default in the JupyterLab file browser anyway. I usually use a system terminal to e.g. cp myrepo/.gitignore gitignore.txt to create a visible copy (somewhere other than the repository folder which you're trying to avoid navigating to!) and then mv gitignore.txt myrepo/.gitignore to overwrite with my edited version

Alternatively (if e.g. it's a folder full of new files that you no longer care about like node_modules) you could just slog through the slowness to delete the problemmatic folder in the UI - but of course the problem would return if you re-created them later without .gitignore.

AWS
EXPERT
Alex_T
answered 2 years ago
  • Thanks a lot Alex! That was the cause! Simply deleting the folder which had thousand of images made the UI responsive again. And then I also noticed that before the deletion the git status bar was permanently in "Refreshing" status. After the deletion, it's quickly toggling between "Refreshing" and "Idle"

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions