Cannot Install external python packages in AWS Glue spark script?

0

Cannot install python packages like opencv-python(cv2),scikit-image in aws glue script. Tried whl and zip files still cant in the job parameters still facing issues in installing the python packages.Enter image description here
Enter image description here
Enter image description here

Any help would be great and also facing issues in converting list of tuples to rdd using parallelize any help in this regard also would be really great

  • There is not enough information here. How did you attempt to include the packages?

asked 2 years ago664 views
1 Answer
0

Hi,

if you can access internet from your job the easiest way to add external libraries is to follow the steps described at this documentation page; while the page refers to Glue 2.0 it is applicable to Glue 3.0 as well.

Under Job parameters, do the following:
For Key, enter --additional-python-modules.
For Value, enter opencv-python, scikit-image

If your job will execute in a VPC with no internet access (not even via a NAT Gateway) please review this blog post.

hope this helps,

AWS
EXPERT
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions