By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

problem with glue and external modules python

0

i have one similar problem,

this error apears here, LAUNCH ERROR | Python Module Installer indicates modules that failed to install, check logs from the PythonModuleInstaller.Please refer logs for details.. Note: This run was executed with Flex execution. Check the logs if run failed due to executor termination.

i was create this file setup.py

Enter image description here

and in job parameter i does this, on the field value i put the link of my s3 bucket with a file .whl Enter image description here

i'll read the documentation but i cant to do execute this file with success, i need help guys!

  • Gonzalo Herreros EXPERT
    9 months ago

    in the log (I think output) there should be details about how pip is installing the modules and dependencies

Topics
Analytics
Tags
AWS Glue
Language
English
David-data-engineer
asked 9 months ago2550 views
5 Answers
3
Accepted Answer

I'd suggest you to go through this re:Post Knowledge Center Article and this blog post.

Another thing to isolate this issue, I'd do is, add modules one by one and keep adding s3 path in the value field. This would help me to find which module is giving me the error.

Some of the modules, you can omit from this list as some of Python modules already provided in AWS Glue out of the box.

This would definitely help. Comment here if you have additional questions.

PS: It'd be helpful, if you can mention the source of wheel package, did you build it or was it directly downloaded and uploaded to s3. Are all the dependencies packaged in the mentioned package.

Abhishek

profile pictureAWS
EXPERT
secondabhi_aws
answered 9 months ago
profile pictureAWS
EXPERT
iBehr
reviewed 9 months ago
1

Hi, Unable to import module in aws glue jobeven after adding details in --additional-python-modules.

--additional-python-modules psycopg2-binary==2.8.6

Tried to specify .s3 path for whl file and tried. still mo luck

any can guide here ?

Vivek
answered 5 months ago
  • rePost-User-3310816
    2 months ago

    same issue

  • Ravi
    a month ago

    vivek, how do you fix --additional-python-modules psycopg2-binary==2.8.6? . i am in same boat.

0

thanks!, i has success, but now i have other problem,

im trying to connect with a api rest

Enter image description here

and this its my code

Enter image description here

i believe that its a some problem with a vpc, but for test i was open all ports inbound and outbound

David-data-engineer
answered 9 months ago
  • secondabhi_aws EXPERT
    9 months ago

    It seems like you have enabled "required connection" in your glue job. Do "Edit job" and remove the required connection if you see one there already. With required connection it keeps the execution environment within your VPC and doesn't allow external connections. If this doesn't apply in your case then look at this re:Post Knowledge Center Article, this would definitely help you to pass this error. Comment here how it goes, happy to help further.

0

For me it turned out to be a network issue during installation (AWS services were reachable). I deployed the Glue Job into the wrong Subnet and I was therefore not able to connect to the NAT Gateways (Internet) and install pip modules (same would apply to your Security Group settings for the Job). Maybe this helps someone else too

etwillbefine
answered 8 months ago
0

Thanks for the response. yes my connection was not able to reach out to internet. I modified the NAT gateway and it worked

Vivek
answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content