- Newest
- Most votes
- Most comments
I'm having this same exact issue and found the culprit, some various environment variables aren't being properly seen as null in Windows when they're being looked up if they aren't set.
Find the GlueKernel.py file for the glue interactive sessions package, in the site-packages folder for your Python environment like:
site-packages\aws_glue_interactive_sessions_kernel\glue_pyspark\GlueKernel.py
Change the function at line 871 to
Every environment variable lookup was always being returned as '${SOME_KEY}' instead of being None.
def _retrieve_os_env_variable(self, key):
_, output = subprocess.getstatusoutput(f"echo ${key}")
if(output != '${}'.format(key)):
return output
else:
return os.environ.get(key)
with the help of a colleague using a Windows laptop we have been able to reproduce it.
thanks Fabrizi, I updated the code in GlueKernel.py, didnt change the terminal still throws the following exception
site-packages\aws_glue_interactive_sessions_kernel\glue_pyspark\GlueKernel.py", line 432, in set_glue_version raise Exception(f"Valid Glue versions are {VALID_GLUE_VERSIONS}") Exception: Valid Glue versions are {'2.0', '3.0'}
did you do have to anything else after updating & just saving the script like re-compiling, etc to make it work ?
Hi,
Have you tried to set the glue version through the spark magic as described here?
%glue_version 2.0
or
%glue_version 3.0
this need to be set in the first cell before you start the session.
I tested this locally and it worked, hope this helps.
the error message pops up on running the Jupyter Notebook command, so the kernel fails to launch (Jupyter notebook immediately disconnects from the Pyspark kernel after launch as a result).
The magic command is only usefuly if the notebook is connected to the Pyspark kernel, which it isn't in this case. Adding glue version to the config files in .aws folder also does not resolve this.
@rePost-User-6963104 , sorry I may have not captured fully your context reading the question. I did the upgrade on my mac and did not see any issue, so I thought you were having the error at session initialization time.
I had to manually hard code the version as 3.0 in the set_glue_version method inside of GlueKernel.py. This got rid of the error, but obviously isn't a great solution. This implementation is pretty buggy.
Thank you for your feedback
Relevant content
- asked 2 years ago
- asked a year ago
- asked a year ago
- AWS OFFICIALUpdated a month ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a month ago
Where do you get this error, in terminal?
yes the error is in the terminal, the pyspark notebook doesn't respond