By using AWS re:Post, you agree to the Terms of Use

Glue Interactive Python Jupyter notebook sessions

1

Hello.

I'm following the AWS docs here [https://docs.aws.amazon.com/glue/latest/dg/interactive-sessions-chapter.html] for developing Glue ETL pyspark jobs on my laptop using remote Glue Spark services in AWS. I get the errors below as soon as I follow the directions to start a session and attempt to actually use the session to do something. Is Glue still a supported service? I see that the documentation is quite old. Error message below.

Traceback (most recent call last):
  File "/Users/xxx/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/aws_glue_interactive_sessions_kernel/glue_python_kernel/GlueKernel.py", line 95, in do_execute
    self.create_session()
  File "/Users/xxx/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/aws_glue_interactive_sessions_kernel/glue_python_kernel/GlueKernel.py", line 518, in create_session
    self.set_session_id(self.glue_client.create_session(
  File "/Users/xxx/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/botocore/client.py", line 657, in __getattr__
    raise AttributeError(
AttributeError: 'Glue' object has no attribute 'create_session'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/xxx/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 357, in dispatch_shell
    await result
  File "/Users/xxx/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 652, in execute_request
    reply_content = await reply_content
  File "/Users/xxx/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/aws_glue_interactive_sessions_kernel/glue_python_kernel/GlueKernel.py", line 97, in do_execute
    self.ipython_display.send_error(f'Exception encountered while creating session: {e} \n')
  File "/Users/xxx/.asdf/installs/python/3.9.7/lib/python3.9/site-packages/hdijupyterutils/ipythondisplay.py", line 30, in send_error
    self._ipython_shell.write_err(u"{}\n".format(error))
AttributeError: 'ZMQInteractiveShell' object has no attribute 'write_err'

Here is my test code....

%iam_role arn:aws:iam::xxxxxx:role/service-role/AWSGlueServiceRole-dev
%session_id test-glue-session1
!jupyter --version
%list_sessions
%connections "dev-poc-glue-aurora-db"
sc = SparkContext()

and the output....

Welcome to the Glue Interactive Sessions Kernel
For more information on available magic commands, please type %help in any new cell.

Current iam_role is None
iam_role has been set to arn:aws:iam::xxxxxxxx:role/service-role/AWSGlueServiceRole-dev.
There is no current session.
Selected Jupyter core packages...
IPython          : 8.0.1
ipykernel        : 6.7.0
ipywidgets       : 7.6.5
jupyter_client   : 7.1.2
jupyter_core     : 4.9.1
jupyter_server   : not installed
jupyterlab       : not installed
nbclient         : 0.5.10
nbconvert        : 6.4.1
nbformat         : 5.1.3
notebook         : 6.4.8
qtconsole        : 5.2.2
traitlets        : 5.1.1
There is no current session.
Connections to be included:
dev-poc-glue-aurora-db
Authenticating with profile=default
glue_role_arn defined by user: arn:aws:iam::xxxxxxxxxxx:role/service-role/AWSGlueServiceRole-dev
Attempting to use existing AssumeRole session credentials.
Trying to create a Glue session for the kernel.
Worker Type: G.1X
Number of Workers: 5
Session ID: AROAJXDQBQMRZML4ALH7A:xxxxxxxxxx-8cff17de-6d96-4021-9f9c-5e6c9f3296b9

thanks so much for your assistance.

asked 8 months ago1324 views
2 Answers
1

Hello,

AWS Glue interactive session is currently in preview, the documentation is reflecting the steps that will need to be run once it is general available.

At the moment, you need still to run a manual step before you start the notebook the first time:

  • add the service model included with the install to botocore.

With your venv active run:

SITE_PACKAGES=$(pip3 show aws-glue-sessions | grep Location | awk '{print $2}')
cp $SITE_PACKAGES/aws_glue_interactive_sessions_kernel/service-2.json $SITE_PACKAGES/botocore/data/glue/2017-03-31/service-2.json

hope this helps

EXPERT
answered 8 months ago
0

Interactive Glue Sessions are now generally available. I made a step-by-step video tutorial on how to configure it to develop locally in pycharm for anyone who wants to follow along. https://youtu.be/04LMQxDxjGM

answered 5 months ago
EXPERT
reviewed 4 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions