By using AWS re:Post, you agree to the Terms of Use

AWS Glue notebook - Kernel fails with Glue version exception

0

I am trying to run the notebooks locally, I followed the instructions provided here (for Windows): https://docs.aws.amazon.com/glue/latest/dg/interactive-sessions.html

I ran the pip3 install --upgrade jupyter boto3 aws-glue-sessions, which upgraded aws-glue-session to version 0.35

but the when starting the notebook the kernel fails to launch and throws the following error,

  File "d:\python38\lib\site-packages\aws_glue_interactive_sessions_kernel\glue_pyspark\GlueKernel.py", line 100, in __init__
    self.set_glue_version(os_env_glue_version)
  File "d:\python38\lib\site-packages\aws_glue_interactive_sessions_kernel\glue_pyspark\GlueKernel.py", line 443, in set_glue_version
    raise Exception(f"Valid Glue versions are {VALID_GLUE_VERSIONS}")
Exception: Valid Glue versions are {'2.0', '3.0'}

Setting the the glue_version = 2.0 in .aws/config and environment variables, does not help either. any help on what could be causing this will be much appreciated!

1 Answer
0

Hi,

Have you tried to set the glue version through the spark magic as described here?

%glue_version 2.0

or

%glue_version 3.0

this need to be set in the first cell before you start the session.

I tested this locally and it worked, hope this helps.

EXPERT
answered 11 days ago
  • the error message pops up on running the Jupyter Notebook command, so the kernel fails to launch (Jupyter notebook immediately disconnects from the Pyspark kernel after launch as a result).

    The magic command is only usefuly if the notebook is connected to the Pyspark kernel, which it isn't in this case. Adding glue version to the config files in .aws folder also does not resolve this.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions