Glue PySpark vs Shell

Question

I zipped my modules into zip file, uploaded to s3 and added to Pyspark and Shell jobs under `Python library path
` parameter:

![Enter image description here](/media/postImages/original/IMtBTEZoIjRwOojwFhhFLl1g)

In both jobs I am using the same import syntax. Pyspark job is working, Python Shell jobs raises error, saying my module not found:

`ModuleNotFoundError: No module named 'expect_multicolumn_values_not_null'
`

Is there a difference between Pyspark and Python Shell modules import? How can I make it work on both?

Ramakrishna Natarajan · Answer

Hi,

Based on my understanding, for the Python Shell jobs, you can consider using this approach [Providing your own Python library](https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html#create-python-extra-library). For your use-case you may need to create an Egg or Whl file.

You can also refer to these posts [How do I use external Python libraries in my AWS Glue 2.0 ETL job?]() and [External python libraries in a AWS Glue python shell job](https://repost.aws/questions/QURu2WtiB2RlKS6Go8m7LFfg/external-python-libraries-in-a-aws-glue-python-shell-job)

Thanks,
Rama

Glue PySpark vs Shell

Añade tu respuesta

Contenido relevante