Help us improve the AWS re:Post Knowledge Center by sharing your feedback in a brief survey. Your input can influence how we create and update our content to better support your AWS journey.
Glue PySpark vs Shell
I zipped my modules into zip file, uploaded to s3 and added to Pyspark and Shell jobs under Python library path parameter:
In both jobs I am using the same import syntax. Pyspark job is working, Python Shell jobs raises error, saying my module not found:
ModuleNotFoundError: No module named 'expect_multicolumn_values_not_null'
Is there a difference between Pyspark and Python Shell modules import? How can I make it work on both?
- Le plus récent
- Le plus de votes
- La plupart des commentaires
Hi,
Based on my understanding, for the Python Shell jobs, you can consider using this approach Providing your own Python library. For your use-case you may need to create an Egg or Whl file.
You can also refer to these posts How do I use external Python libraries in my AWS Glue 2.0 ETL job? and External python libraries in a AWS Glue python shell job
Thanks, Rama
Contenus pertinents
- demandé il y a 10 mois
- demandé il y a 10 mois
- demandé il y a 3 ans
- AWS OFFICIELA mis à jour il y a 4 ans
