Using External Python Packages on AWS Glue.


I zipped my custom created python script to my S3 bucket. I updated the bucket path in my job configuration. Even after doing this I am not able to import my script to my Glue Job.

Below is error displayed:

Traceback (most recent call last): File "/tmp/", line 211, in <module> runpy.run_path(temp_file_path, run_name='main') File "/usr/local/lib/python3.6/", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/usr/local/lib/python3.6/", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/usr/local/lib/python3.6/", line 85, in _run_code exec(code, run_globals) File "/tmp/glue-python-scripts-fer1ln57/", line 2, in <module> ModuleNotFoundError: No module named 'test'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/tmp/", line 230, in <module> raise e_type(e_value).with_traceback(new_stack) File "/tmp/glue-python-scripts-fer1ln57/", line 2, in <module> ModuleNotFoundError: No module named 'test'

My code details:

  • It is just a function which is printing a string.

This is my test script.

My main goal is to use external packages to achieve my ETL needs.

I tried zipping python files folder which contain python files even uploaded single python file as well but, nothing seem to work for me.

asked 3 years ago3138 views
1 Answer

Hi @Nihal-Rainu,

Refer the following links to resolve the error - the first link is related to AWS Glue 2.0 and the second link helps address the problem for AWS Glue 1.0 or .9...

Also, the following link describes how AWS Glue lets you install additional Python modules and libraries for use with AWS Glue ETL...

And finally the following link talks about how you can use a Python shell job to run Python scripts (that are compatible with Python 2.7 or Python 3.6) as a shell in AWS Glue...

Hope this helps.

answered 3 years ago
reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions