AWS Glue Pythonshell 3.6 - custom egg is not supported ?

Hi,

Egg files are supported for both Pythonshell 2.7 and 3.6.

Can you please confirm if the Python Version used to create egg file was 3.6? I suspect that the version used might be 3.7 and due to some incompatibility issue, it would have failed with Pythonshell 3.6. However, to figure out the actual cause of failure, we will need to look into the job run logs.

Looking forward to hear from you.

Yes, my version was set to 3.7. I switched the version to 3.6 and everything worked. Thank you!

Hi Team,

I am facing the same issue in Glue python shell job. I have followed the steps which is given in below AWS link to generate dependent python egg files and wheel files. I have created egg wheel files & using python 3.6 and uploaded to s3 and given to lib path to glue python shell jobs. But still job are getting failed with "Module not found error". Could you please help me to resolve the issue.

https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html#python-shell-supported-library

Cloudwatch log:
Installed /glue/lib/installation/pymysql-0.9.3-py3.6.egg
Processing dependencies for pymysql==0.9.3
Finished processing dependencies for pymysql==0.9.3
Processing ./glue-python-libs-kcshw19b/pymysql-0.9.3-py3-none-any.whl
Installing collected packages: pymysql
Successfully installed pymysql-0.9.3
Requirement already satisfied: pymysql in /glue/lib/installation (0.9.3)

cloudwatrch error log:
ModuleNotFoundError: No module named 'pymysql'