Uncaught exception in ZMQStream callback -- trying to run Jupyter notebook with Julia kernel in SageMaker

Thanks for raising this again!

So as mentioned on the other question, I tentatively believe this to be caused by this open IJulia issue... And there I see one participant commented they managed to resolve the issue by not installing Julia inside conda.

Happy to share I seem to have been able to reproduce this success with the following steps... :D

1. Instead of creating a conda env and installing Julia through it as advised by the whitepaper (which would of course be a good practice for proper environment separation, if it weren't for this issue), wget the generic Linux x86 64-bit (glibc) tarball from the Julia downloads page https://julialang.org/downloads/ (for example, https://julialang-s3.julialang.org/bin/linux/x64/1.7/julia-1.7.2-linux-x86_64.tar.gz)
   • (Because uname -a shows us that our NBI is running in x86 64-bit Linux, and a couple of hacky checking methods I found online seemed to suggest it's a glibc setup rather than musl)
   • Don't forget to cd SageMaker in terminal before your wget, so the file actually downloads to somewhere you can see it in JupyterLab
2. (Verify the checksum of the downloaded file first, as good security practice and then) Untar the bundle with tar -xvf {YourFileName.tar.gz}. You should see a folder something like julia-1.7.2 containing (amongst other things) a bin subfolder with a julia executable in it.
3. Make julia executable visible in your system PATH
   • There are many different ways you might want to set this up, depending what you want on persistent EBS storage vs ephemeral, whether you want to use multiple versions of julia in parallel, and whether you have preferences about where in the filesystem the application should reside.
   • For this particular test I chose to cp -R ./julia-1.7.2 /usr/local/julia (copy Julia to a folder where I wouldn't accidentally edit it) and then sudo ln -s /usr/local/julia/bin/julia /bin/julia (create a julia symlink to the main executable inside the /bin folder, which is already on PATH).
4. Start julia from your terminal (julia) and then install & configure IJulia as per the original instructions:

using Pkg
Pkg.add("IJulia")
using IJulia
jupyterlab(detached=true)
# I got an error on this last command but it still seemed to work?

5. Verify that you should now have a julia entry in your Jupyter kernels folder. I copied out the contents into my JupyterLab working folder to inspect it: cp -R ~/.local/share/jupyter/kernels/julia-1.7 ./julia-1.7-copy. It should contain a kernel.json.
6. If all is well (and it was for me) you should now see a "Julia 1.7.2" (or similar version) kernel on your launcher and be able to launch a notebook successfully and run commands (🎉)

What about productionization?

So of course this installation is a bit of a hacky workaround for whatever underlying issue is causing Julia to not run properly inside conda... I'd really prefer to see a direct resolution for that problem if possible. The procedure described here won't persist between Notebook Instance stop/starts (at which time everything outside of /home/ec2-user/SageMaker EBS mount is reset), so you'd probably want to explore automating the setup with a lifecycle configuration script.

I see by default that Julia packages (including IJulia) are installed to /home/ec2-user/.julia/packages/ (which is outside the SageMaker mount), so may be interesting to explore configuration options for moving .julia under the SageMaker folder for persistence. Files and folders beginning with a dot are hidden in JupyterLab file tree, which is worth remembering if you explore customizing the setup.