make julia notebooks work in SageMaker

Sorry this is not a full working solution, but too much for a comment & hopefully will still be useful to you:

(Assuming you're talking here about SageMaker Notebook Instances rather than SageMaker Studio, same as the whitepaper; that you're trying to install the latest version of Julia from conda rather than v1.0.3 explicitly specified in the whitepaper, currently 1.7.1)

The first thing to point out is that you should be able to debug this via the notebook's logs: Either click the "View logs" link on the notebook's detail page in Amazon SageMaker console, or look in the Amazon CloudWatch console for log group /aws/sagemaker/NotebookInstances, stream {YOUR-NBI-NAME}/jupyter.log.

Following through the whitepaper instructions myself (on an notebook-al2-v1 notebook instance), the errors I saw preventing the kernel from loading were like:

ArgumentError: Package IJulia not found in current path:
- Run `import Pkg; Pkg.add("IJulia")` to install the IJulia package.

Looking at ~/.local/share/jupyter/kernels/julia-1.7/kernel.json, I saw the created kernel was defined as follows:

{
  "display_name": "Julia 1.7.1",
  "argv": [
    "/home/ec2-user/anaconda3/envs/julia/bin/julia",
    "-i",
    "--color=yes",
    "--project=@.",
    "/home/ec2-user/anaconda3/envs/julia/share/julia/packages/IJulia/e8kqU/src/kernel.jl",
    "{connection_file}"
  ],
  "language": "julia",
  "env": {},
  "interrupt_mode": "signal"
}

If we run julia from within the julia conda environment, using IJulia works no problem... However, if you source activate JupyterSystemEnv from the terminal - you can still run /home/ec2-user/anaconda3/envs/julia/bin/julia from the terminal but the interpreter will not think the IJulia package is installed... I think this is closer to what the above kernel definition is doing (NBI JupyterServer itself runs in this system conda env).

I tried a couple of hacky solutions:

1. Simply start Julia in JupyterSystemEnv as above and Pkg.add("IJulia") to install it there
2. Copy the setup you'll see in the R kernel, /home/ec2-user/.local/share/jupyter/kernels/ir - where kernel.json points to a run.sh script which first activates the target conda environment and then runs the interpreter

kernel.json (with a different display name & file path to visually confirm JupyterLab has picked the new one up):

{
  "display_name": "Julia Fix",
  "argv": [
    "/home/ec2-user/.local/share/jupyter/kernels/julia-fix/run.sh",
    "{connection_file}"
  ],
  "language": "julia",
  "env": {},
  "interrupt_mode": "signal"
}

run.sh (remember to chmod +x this file to avoid 500 server errors as Jupyter can't execute it)

#!/bin/bash

source activate julia
/home/ec2-user/anaconda3/envs/julia/bin/julia -i --project=@. /home/ec2-user/anaconda3/envs/julia/share/julia/packages/IJulia/e8kqU/src/kernel.jl $1

HOWEVER, unfortunately both approaches yield a ZMQStream Invalid Signature error, which looks to me like this open IJulia issue. I see speculation there that Julia isn't playing nice with conda, but am not deep enough with Julia to know if that's really the root cause or how best to mitigate it if so.

Maybe you could install Julia itself outside of conda (conda deactivate in terminal to check you're in base environment) and, if still getting the signature error, use a run.sh kernel script to ensure Jupyter also runs the command outside of conda rather than in the JupyterSystemEnv? Or perhaps there's some other cause for the signature issue that can be resolved while keeping conda environments set up...

You could also look into the SageMaker Studio Custom Image Samples, instead of Notebook Instances, since Studio kernels are isolated by full container images rather than conda? There is an example image there for Julia (v1.5), although it's not been updated in some time so of course could have some issues of its own.

Note that Alex_T found a (hacky) solution to the ZMQStream problem here:

https://www.repost.aws/questions/QUM2y8-rjDS1Wp5LUkdLMhyA/uncaught-exception-in-zmq-stream-callback-trying-to-run-jupyter-notebook-with-julia-kernel-in-sage-maker

and got the Jupyter notebook with Julia kernel to work in SageMaker.