%additional_python_modules failure in glue notebook


while additional_python_modules work in Glue version 4.0, but since I cannot change the glue version in glue notebook (Glue notebook supports 2.0 and 3.0 only), the whl file pointing to s3 path fails to import in glue notebook. Its a deceiving message when it says it actually used the whl file. Can you fix that in glue notebook or remove it from %help. I learnt it the hard way.

asked 7 days ago42 views
3 Answers

Hello, I see that you are unable to import the python package into your Glue notebook. You tried to install the dependency from whl file stored in s3 bucket. I have tried it on my test environment and I was able to import my module using whl file stored in s3 bucket. Here are the steps that I followed:

  1. For testing purpose, I uploaded this whl file into my s3 bucket.
  2. Next, I started a Glue notebook and then I used a magic to give the s3 bucket path of my wheel file. The magic command was something like this:
%additional_python_modules <my-s3-uri>
  1. I executed this and then my session got created. When I tried to import psycopg2 module, it was working fine for me.

Therefore, I request you to verify the following things on your end:

  • Do check whether or not you have followed the above steps.
  • Please do note that once the session is created, it is not possible to run the magic command. You will have to stop and then restart the notebook to run a magic (%additional_python_modules in your case). These magic commands will only run at the time of session creation.
  • Confirm whether the whl file that you are trying to install is compatible with the Glue version you are using.

If you are still facing the same issue, then please share me the steps you were trying to follow along with the error that you were facing.

profile picture
answered 7 days ago

Enter image description here

Screenshot attached

answered 6 days ago

Hey Chaitu, Thank you for the response. Appreciate it. However as an end user had the notebook errored out while loading the library, I would have saved much time and pain that i wasted in troubleshooting. I am sure as you won't expect customers to be aware of these nuances by expecting them to read all the documentation. Could have been avoided by failing fast. Now this has opened a pandora box. I have to downgrade all my glue notebooks from 4.0 to 3.0 so that we maintain same package versions and compatibility in dev notebooks and prod glue jobs. Please pass this feedback to the maintainers.

answered 5 days ago
  • I will pass on the feedback as suggested by you. So, you could have tried it with the normal Glue job set to version 3.0 and then observed the "error logs" for the job. It will show the error message if it fails to install the python library. In fact, I was able to find this out by the same way. Having said that, it could have been much easier & time saving for you if the error was displayed on the notebook itself. I will convey this to the relevant team.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions