%additional_python_modules failure in glue notebook

0

while additional_python_modules work in Glue version 4.0, but since I cannot change the glue version in glue notebook (Glue notebook supports 2.0 and 3.0 only), the whl file pointing to s3 path fails to import in glue notebook. Its a deceiving message when it says it actually used the whl file. Can you fix that in glue notebook or remove it from %help. I learnt it the hard way.

질문됨 일 년 전570회 조회
3개 답변
0

Hello, I see that you are unable to import the python package into your Glue notebook. You tried to install the dependency from whl file stored in s3 bucket. I have tried it on my test environment and I was able to import my module using whl file stored in s3 bucket. Here are the steps that I followed:

  1. For testing purpose, I uploaded this whl file into my s3 bucket.
  2. Next, I started a Glue notebook and then I used a magic to give the s3 bucket path of my wheel file. The magic command was something like this:
%additional_python_modules <my-s3-uri>
  1. I executed this and then my session got created. When I tried to import psycopg2 module, it was working fine for me.

Therefore, I request you to verify the following things on your end:

  • Do check whether or not you have followed the above steps.
  • Please do note that once the session is created, it is not possible to run the magic command. You will have to stop and then restart the notebook to run a magic (%additional_python_modules in your case). These magic commands will only run at the time of session creation.
  • Confirm whether the whl file that you are trying to install is compatible with the Glue version you are using.

If you are still facing the same issue, then please share me the steps you were trying to follow along with the error that you were facing.

profile pictureAWS
지원 엔지니어
Chaitu
답변함 일 년 전
0

Enter image description here

Screenshot attached

답변함 일 년 전
0

Hey Chaitu, Thank you for the response. Appreciate it. However as an end user had the notebook errored out while loading the library, I would have saved much time and pain that i wasted in troubleshooting. I am sure as you won't expect customers to be aware of these nuances by expecting them to read all the documentation. Could have been avoided by failing fast. Now this has opened a pandora box. I have to downgrade all my glue notebooks from 4.0 to 3.0 so that we maintain same package versions and compatibility in dev notebooks and prod glue jobs. Please pass this feedback to the maintainers.

Sunil
답변함 일 년 전
  • I will pass on the feedback as suggested by you. So, you could have tried it with the normal Glue job set to version 3.0 and then observed the "error logs" for the job. It will show the error message if it fails to install the python library. In fact, I was able to find this out by the same way. Having said that, it could have been much easier & time saving for you if the error was displayed on the notebook itself. I will convey this to the relevant team.

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인