Using External Python Packages on AWS Glue.

0

I zipped my custom created python script to my S3 bucket. I updated the bucket path in my job configuration. Even after doing this I am not able to import my script to my Glue Job.

Below is error displayed:

Traceback (most recent call last): File "/tmp/runscript.py", line 211, in <module> runpy.run_path(temp_file_path, run_name='main') File "/usr/local/lib/python3.6/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/usr/local/lib/python3.6/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/tmp/glue-python-scripts-fer1ln57/DocumentDB-Glue-ETL.py.py", line 2, in <module> ModuleNotFoundError: No module named 'test'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/tmp/runscript.py", line 230, in <module> raise e_type(e_value).with_traceback(new_stack) File "/tmp/glue-python-scripts-fer1ln57/DocumentDB-Glue-ETL.py.py", line 2, in <module> ModuleNotFoundError: No module named 'test'

My code details:

  • It is just a function which is printing a string.

This is my test script.

My main goal is to use external packages to achieve my ETL needs.

I tried zipping python files folder which contain python files even uploaded single python file as well but, nothing seem to work for me.

gefragt vor 2 Jahren2994 Aufrufe
1 Antwort
0

Hi @Nihal-Rainu,

Refer the following links to resolve the error - the first link is related to AWS Glue 2.0 and the second link helps address the problem for AWS Glue 1.0 or .9...

https://aws.amazon.com/premiumsupport/knowledge-center/glue-version2-external-python-libraries/ https://aws.amazon.com/premiumsupport/knowledge-center/glue-job-use-external-python-libraries/

Also, the following link describes how AWS Glue lets you install additional Python modules and libraries for use with AWS Glue ETL... https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html

And finally the following link talks about how you can use a Python shell job to run Python scripts (that are compatible with Python 2.7 or Python 3.6) as a shell in AWS Glue...

https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html

Hope this helps.

AWS
beantwortet vor 2 Jahren
AWS
EXPERTE
überprüft vor 2 Jahren

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen