Cannot Install external python packages in AWS Glue spark script?

Question

Cannot install python packages like opencv-python(cv2),scikit-image in aws glue script. Tried whl and zip files still cant in the job parameters still facing issues in installing the python packages.![![![Enter image description here](/media/postImages/original/IM_C-S_459TLKK07sasau9Qw)
Enter image description here](/media/postImages/original/IMcpdik4-8TguwsfdBvv4piA)
Enter image description here](/media/postImages/original/IMpmpJs5EkTnCi8DdIsijvRg)

Any help would be great and also facing issues in converting list of tuples to rdd using parallelize any help in this regard also would be really great

Answer

Hi,

if you can access internet from  your job the easiest way to add external libraries is to follow the steps described [at this documentation page](https://aws.amazon.com/premiumsupport/knowledge-center/glue-version2-external-python-libraries/); while the page refers to Glue 2.0 it is applicable to Glue 3.0 as well.

```
Under Job parameters, do the following:
For Key, enter --additional-python-modules.
For Value, enter opencv-python, scikit-image
```

If your job will execute in a VPC with no internet access (not even via a NAT Gateway) please review [this blog post](https://aws.amazon.com/blogs/big-data/building-python-modules-from-a-wheel-for-spark-etl-workloads-using-aws-glue-2-0/).

hope this helps,

Cannot Install external python packages in AWS Glue spark script?

Contenuto pertinente