By using AWS re:Post, you agree to the Terms of Use

AWS Glue Pythonshell 3.6 - custom egg is not supported ?

0

Created my custom package and build an egg. The egg (with 3rd party libraries) was referenced and installed properly using the Glue Python 2 Shell Job. If I build a similar egg using python 3 and reference in Glue Shell Job Python 3 (with 3rd party libraries) then it doesn't get installed.

Does python3 shell support third-party or custom eggs?

Python 2 direction for providing own python library can be found here, https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html

asked 3 years ago110 views
3 Answers
0

Hi,

Egg files are supported for both Pythonshell 2.7 and 3.6.

Can you please confirm if the Python Version used to create egg file was 3.6? I suspect that the version used might be 3.7 and due to some incompatibility issue, it would have failed with Pythonshell 3.6. However, to figure out the actual cause of failure, we will need to look into the job run logs.

Looking forward to hear from you.

answered 3 years ago
0

Yes, my version was set to 3.7. I switched the version to 3.6 and everything worked. Thank you!

answered 3 years ago
0

Hi Team,

I am facing the same issue in Glue python shell job. I have followed the steps which is given in below AWS link to generate dependent python egg files and wheel files. I have created egg wheel files & using python 3.6 and uploaded to s3 and given to lib path to glue python shell jobs. But still job are getting failed with "Module not found error". Could you please help me to resolve the issue.

https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html#python-shell-supported-library

Cloudwatch log:
Installed /glue/lib/installation/pymysql-0.9.3-py3.6.egg
Processing dependencies for pymysql==0.9.3
Finished processing dependencies for pymysql==0.9.3
Processing ./glue-python-libs-kcshw19b/pymysql-0.9.3-py3-none-any.whl
Installing collected packages: pymysql
Successfully installed pymysql-0.9.3
Requirement already satisfied: pymysql in /glue/lib/installation (0.9.3)

cloudwatrch error log:
ModuleNotFoundError: No module named 'pymysql'

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions