AWS Glue notebook - Kernel fails with Glue version exception

Question

I am trying to run the notebooks locally,  I followed the instructions provided here (for Windows): https://docs.aws.amazon.com/glue/latest/dg/interactive-sessions.html

I ran the pip3 install --upgrade jupyter boto3 aws-glue-sessions, which upgraded aws-glue-session to version 0.35

but the when starting the notebook the kernel fails to launch and throws the following error, 
```
  File "d:\python38\lib\site-packages\aws_glue_interactive_sessions_kernel\glue_pyspark\GlueKernel.py", line 100, in __init__
    self.set_glue_version(os_env_glue_version)
  File "d:\python38\lib\site-packages\aws_glue_interactive_sessions_kernel\glue_pyspark\GlueKernel.py", line 443, in set_glue_version
    raise Exception(f"Valid Glue versions are {VALID_GLUE_VERSIONS}")
Exception: Valid Glue versions are {'2.0', '3.0'}
```
Setting the the glue_version = 2.0 in .aws/config and environment variables, does not help either. any help on what could be causing this will be much appreciated!

Answer

I'm having this same exact issue and found the culprit, some various environment variables aren't being properly seen as null in Windows when they're being looked up if they aren't set.

Find the GlueKernel.py file for the glue interactive sessions package, in the site-packages folder for your Python environment like:

site-packages\aws_glue_interactive_sessions_kernel\glue_pyspark\GlueKernel.py

Change the function at line 871 to

Every environment variable lookup was always being returned as '${SOME_KEY}' instead of being None.
```
def _retrieve_os_env_variable(self, key):
    _, output = subprocess.getstatusoutput(f"echo ${key}")
    if(output != '${}'.format(key)):
            return output
        else:
            return os.environ.get(key)
```

Answer

Hi,

Have you tried to set the glue version through the spark magic as described [here](https://docs.aws.amazon.com/glue/latest/dg/interactive-sessions-magics.html)?

```
%glue_version 2.0
```

or

```
%glue_version 3.0
```
this need to be set in the first cell before you start the session.

I tested this locally and it worked, hope this helps.

Answer

I had to manually hard code the version as 3.0 in the set_glue_version method inside of GlueKernel.py.  This got rid of the error, but obviously isn't a great solution. This implementation is pretty buggy.

AWS Glue notebook - Kernel fails with Glue version exception

Relevant content