Uncaught exception in ZMQStream callback -- trying to run Jupyter notebook with Julia kernel in SageMaker

Question

I want to run a Jupyter notebook with Julia kernel in Amazon SageMaker. The Julia 1.7.1 icon shows up in the Jupyterlab launcher and accepts the kernel on launch, but then the kernel dies immediately after launch (it never works). I have posted about this here

https://www.repost.aws/questions/QU2PXu3tbpQ7-V2OTlD_I07Q/make-julia-notebooks-work-in-sage-maker

and Alex_T has made some very good suggestions, but even they get stumped at this point. Log info:

```
[E 00:25:58.760 NotebookApp] Uncaught exception in ZMQStream callback
    Traceback (most recent call last):
      File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/zmq/eventloop/zmqstream.py", line 431, in _run_callback
        callback(*args, **kwargs)
      File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/notebook/services/kernels/kernelmanager.py", line 391, in record_activity
        msg = session.deserialize(fed_msg_list)
      File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/jupyter_client/session.py", line 929, in deserialize
        raise ValueError("Invalid Signature: %r" % signature)
    ValueError: Invalid Signature: b'ec1d1093f3b6505658469b860c203f696bc39cf8fcea1672cb55802fc57592eef57b8db0b5cb603d1bcada6f41060f0819f6002a7a31f309fbaa1a701cc13f5b'
```

There are some posts from 2018 recommending updating tornado and ipykernel, but 4 years later this should hardly apply (and I tried it, and it didn't work). Any suggestions? What could be going wrong here? Everything else seems to be in place. There is a white paper on running a Jupyter notebook with Julia kernel in SageMaker here [https://d1.awsstatic.com/whitepapers/julia-on-sagemaker.pdf?did=wp_card&trk=wp_card], but it is incomplete, as you can see in the other post.

Answer

Thanks for raising this again!

So as mentioned on the [other question](https://repost.aws/questions/QU2PXu3tbpQ7-V2OTlD_I07Q/make-julia-notebooks-work-in-sage-maker), I tentatively believe this to be caused by [this open IJulia issue](https://github.com/JuliaLang/IJulia.jl/issues/1024)... And there I see one participant [commented](https://github.com/JuliaLang/IJulia.jl/issues/1024#issuecomment-996588668) they managed to resolve the issue by *not installing Julia inside conda*.

Happy to share I seem to have been able to reproduce this success with the following steps... :D

1. **Instead** of creating a conda env and installing Julia through it as advised by the whitepaper (which would of course be a good practice for proper environment separation, if it weren't for this issue), `wget` the generic Linux x86 64-bit (glibc) tarball from the Julia downloads page https://julialang.org/downloads/ (for example, `https://julialang-s3.julialang.org/bin/linux/x64/1.7/julia-1.7.2-linux-x86_64.tar.gz`)
    - (Because `uname -a` shows us that our NBI is running in x86 64-bit Linux, and a couple of hacky checking methods I found online seemed to suggest it's a glibc setup rather than musl)
    - Don't forget to `cd SageMaker` in terminal before your `wget`, so the file actually downloads to somewhere you can see it in JupyterLab
2. (Verify the checksum of the downloaded file first, as good security practice and then) Untar the bundle with `tar -xvf {YourFileName.tar.gz}`. You should see a folder something like `julia-1.7.2` containing (amongst other things) a `bin` subfolder with a `julia` executable in it.
3. Make julia executable visible in your system PATH
    - There are many different ways you might want to set this up, depending what you want on persistent EBS storage vs ephemeral, whether you want to use multiple versions of julia in parallel, and whether you have preferences about where in the filesystem the application should reside.
    - For this particular test I chose to `cp -R ./julia-1.7.2 /usr/local/julia` (copy Julia to a folder where I wouldn't accidentally edit it) and then `sudo ln -s /usr/local/julia/bin/julia /bin/julia` (create a `julia` symlink to the main executable inside the `/bin` folder, which is already on PATH).
4. Start julia from your terminal (`julia`) and then install & configure IJulia as per the original instructions:
```
using Pkg
Pkg.add("IJulia")
using IJulia
jupyterlab(detached=true)
# I got an error on this last command but it still seemed to work?
```
5. Verify that you should now have a julia entry in your Jupyter kernels folder. I copied out the contents into my JupyterLab working folder to inspect it: `cp -R ~/.local/share/jupyter/kernels/julia-1.7 ./julia-1.7-copy`. It should contain a `kernel.json`.
6. If all is well (and it was for me) you should now see a "Julia 1.7.2" (or similar version) kernel on your launcher and be able to launch a notebook successfully and run commands (🎉)

### What about productionization?

So of course this installation is a bit of a hacky workaround for whatever underlying issue is causing Julia to not run properly inside conda... I'd really prefer to see a direct resolution for that problem if possible. The procedure described here won't persist between Notebook Instance stop/starts (at which time everything outside of `/home/ec2-user/SageMaker` EBS mount is reset), so you'd probably want to explore automating the setup with a [lifecycle configuration script](https://docs.aws.amazon.com/sagemaker/latest/dg/notebook-lifecycle-config.html).

I see by default that Julia packages (including IJulia) are installed to `/home/ec2-user/.julia/packages/` (which is outside the `SageMaker` mount), so may be interesting to explore configuration options for moving `.julia` under the `SageMaker` folder for persistence. Files and folders beginning with a dot are hidden in JupyterLab file tree, which is worth remembering if you explore customizing the setup.

Uncaught exception in ZMQStream callback -- trying to run Jupyter notebook with Julia kernel in SageMaker

What about productionization?

Relevant content