Using External Python Packages on AWS Glue.

0

I zipped my custom created python script to my S3 bucket. I updated the bucket path in my job configuration. Even after doing this I am not able to import my script to my Glue Job.

Below is error displayed:

Traceback (most recent call last): File "/tmp/runscript.py", line 211, in <module> runpy.run_path(temp_file_path, run_name='main') File "/usr/local/lib/python3.6/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/usr/local/lib/python3.6/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/tmp/glue-python-scripts-fer1ln57/DocumentDB-Glue-ETL.py.py", line 2, in <module> ModuleNotFoundError: No module named 'test'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/tmp/runscript.py", line 230, in <module> raise e_type(e_value).with_traceback(new_stack) File "/tmp/glue-python-scripts-fer1ln57/DocumentDB-Glue-ETL.py.py", line 2, in <module> ModuleNotFoundError: No module named 'test'

My code details:

  • It is just a function which is printing a string.

This is my test script.

My main goal is to use external packages to achieve my ETL needs.

I tried zipping python files folder which contain python files even uploaded single python file as well but, nothing seem to work for me.

asked 2 years ago2930 views
1 Answer
0

Hi @Nihal-Rainu,

Refer the following links to resolve the error - the first link is related to AWS Glue 2.0 and the second link helps address the problem for AWS Glue 1.0 or .9...

https://aws.amazon.com/premiumsupport/knowledge-center/glue-version2-external-python-libraries/ https://aws.amazon.com/premiumsupport/knowledge-center/glue-job-use-external-python-libraries/

Also, the following link describes how AWS Glue lets you install additional Python modules and libraries for use with AWS Glue ETL... https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html

And finally the following link talks about how you can use a Python shell job to run Python scripts (that are compatible with Python 2.7 or Python 3.6) as a shell in AWS Glue...

https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html

Hope this helps.

AWS
answered 2 years ago
AWS
EXPERT
reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions