Uncaught exception in ZMQStream callback -- trying to run Jupyter notebook with Julia kernel in SageMaker

1

I want to run a Jupyter notebook with Julia kernel in Amazon SageMaker. The Julia 1.7.1 icon shows up in the Jupyterlab launcher and accepts the kernel on launch, but then the kernel dies immediately after launch (it never works). I have posted about this here

https://www.repost.aws/questions/QU2PXu3tbpQ7-V2OTlD_I07Q/make-julia-notebooks-work-in-sage-maker

and Alex_T has made some very good suggestions, but even they get stumped at this point. Log info:

[E 00:25:58.760 NotebookApp] Uncaught exception in ZMQStream callback
    Traceback (most recent call last):
      File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/zmq/eventloop/zmqstream.py", line 431, in _run_callback
        callback(*args, **kwargs)
      File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/notebook/services/kernels/kernelmanager.py", line 391, in record_activity
        msg = session.deserialize(fed_msg_list)
      File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/jupyter_client/session.py", line 929, in deserialize
        raise ValueError("Invalid Signature: %r" % signature)
    ValueError: Invalid Signature: b'ec1d1093f3b6505658469b860c203f696bc39cf8fcea1672cb55802fc57592eef57b8db0b5cb603d1bcada6f41060f0819f6002a7a31f309fbaa1a701cc13f5b'

There are some posts from 2018 recommending updating tornado and ipykernel, but 4 years later this should hardly apply (and I tried it, and it didn't work). Any suggestions? What could be going wrong here? Everything else seems to be in place. There is a white paper on running a Jupyter notebook with Julia kernel in SageMaker here [https://d1.awsstatic.com/whitepapers/julia-on-sagemaker.pdf?did=wp_card&trk=wp_card], but it is incomplete, as you can see in the other post.

질문됨 2년 전1410회 조회
1개 답변
1

Thanks for raising this again!

So as mentioned on the other question, I tentatively believe this to be caused by this open IJulia issue... And there I see one participant commented they managed to resolve the issue by not installing Julia inside conda.

Happy to share I seem to have been able to reproduce this success with the following steps... :D

  1. Instead of creating a conda env and installing Julia through it as advised by the whitepaper (which would of course be a good practice for proper environment separation, if it weren't for this issue), wget the generic Linux x86 64-bit (glibc) tarball from the Julia downloads page https://julialang.org/downloads/ (for example, https://julialang-s3.julialang.org/bin/linux/x64/1.7/julia-1.7.2-linux-x86_64.tar.gz)
    • (Because uname -a shows us that our NBI is running in x86 64-bit Linux, and a couple of hacky checking methods I found online seemed to suggest it's a glibc setup rather than musl)
    • Don't forget to cd SageMaker in terminal before your wget, so the file actually downloads to somewhere you can see it in JupyterLab
  2. (Verify the checksum of the downloaded file first, as good security practice and then) Untar the bundle with tar -xvf {YourFileName.tar.gz}. You should see a folder something like julia-1.7.2 containing (amongst other things) a bin subfolder with a julia executable in it.
  3. Make julia executable visible in your system PATH
    • There are many different ways you might want to set this up, depending what you want on persistent EBS storage vs ephemeral, whether you want to use multiple versions of julia in parallel, and whether you have preferences about where in the filesystem the application should reside.
    • For this particular test I chose to cp -R ./julia-1.7.2 /usr/local/julia (copy Julia to a folder where I wouldn't accidentally edit it) and then sudo ln -s /usr/local/julia/bin/julia /bin/julia (create a julia symlink to the main executable inside the /bin folder, which is already on PATH).
  4. Start julia from your terminal (julia) and then install & configure IJulia as per the original instructions:
using Pkg
Pkg.add("IJulia")
using IJulia
jupyterlab(detached=true)
# I got an error on this last command but it still seemed to work?
  1. Verify that you should now have a julia entry in your Jupyter kernels folder. I copied out the contents into my JupyterLab working folder to inspect it: cp -R ~/.local/share/jupyter/kernels/julia-1.7 ./julia-1.7-copy. It should contain a kernel.json.
  2. If all is well (and it was for me) you should now see a "Julia 1.7.2" (or similar version) kernel on your launcher and be able to launch a notebook successfully and run commands (🎉)

What about productionization?

So of course this installation is a bit of a hacky workaround for whatever underlying issue is causing Julia to not run properly inside conda... I'd really prefer to see a direct resolution for that problem if possible. The procedure described here won't persist between Notebook Instance stop/starts (at which time everything outside of /home/ec2-user/SageMaker EBS mount is reset), so you'd probably want to explore automating the setup with a lifecycle configuration script.

I see by default that Julia packages (including IJulia) are installed to /home/ec2-user/.julia/packages/ (which is outside the SageMaker mount), so may be interesting to explore configuration options for moving .julia under the SageMaker folder for persistence. Files and folders beginning with a dot are hidden in JupyterLab file tree, which is worth remembering if you explore customizing the setup.

AWS
전문가
Alex_T
답변함 2년 전
  • You are a hero, Alex_T! It worked perfectly for me, and I didn't even get an error running

    jupyterlab(detached=true)

    Thank you!

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠