Usando AWS re:Post, accetti AWS re:Post Termini di utilizzo
AWS re:Postre:Post

problem with glue and external modules python

0

i have one similar problem,

this error apears here, LAUNCH ERROR | Python Module Installer indicates modules that failed to install, check logs from the PythonModuleInstaller.Please refer logs for details.. Note: This run was executed with Flex execution. Check the logs if run failed due to executor termination.

i was create this file setup.py

Enter image description here

and in job parameter i does this, on the field value i put the link of my s3 bucket with a file .whl Enter image description here

i'll read the documentation but i cant to do execute this file with success, i need help guys!

  • Gonzalo Herreros ESPERTO
    10 mesi fa

    in the log (I think output) there should be details about how pip is installing the modules and dependencies

Argomenti
Analisi
Tag
AWS Glue
Lingua
English
David-data-engineer
posta 10 mesi fa2779 visualizzazioni
5 Risposte
3
Risposta accettata

I'd suggest you to go through this re:Post Knowledge Center Article and this blog post.

Another thing to isolate this issue, I'd do is, add modules one by one and keep adding s3 path in the value field. This would help me to find which module is giving me the error.

Some of the modules, you can omit from this list as some of Python modules already provided in AWS Glue out of the box.

This would definitely help. Comment here if you have additional questions.

PS: It'd be helpful, if you can mention the source of wheel package, did you build it or was it directly downloaded and uploaded to s3. Are all the dependencies packaged in the mentioned package.

Abhishek

profile pictureAWS
ESPERTO
secondabhi_aws
con risposta 10 mesi fa
profile pictureAWS
ESPERTO
iBehr
verificato 10 mesi fa
1

Hi, Unable to import module in aws glue jobeven after adding details in --additional-python-modules.

--additional-python-modules psycopg2-binary==2.8.6

Tried to specify .s3 path for whl file and tried. still mo luck

any can guide here ?

Vivek
con risposta 6 mesi fa
  • rePost-User-3310816
    3 mesi fa

    same issue

  • Ravi
    2 mesi fa

    vivek, how do you fix --additional-python-modules psycopg2-binary==2.8.6? . i am in same boat.

0

thanks!, i has success, but now i have other problem,

im trying to connect with a api rest

Enter image description here

and this its my code

Enter image description here

i believe that its a some problem with a vpc, but for test i was open all ports inbound and outbound

David-data-engineer
con risposta 10 mesi fa
  • secondabhi_aws ESPERTO
    10 mesi fa

    It seems like you have enabled "required connection" in your glue job. Do "Edit job" and remove the required connection if you see one there already. With required connection it keeps the execution environment within your VPC and doesn't allow external connections. If this doesn't apply in your case then look at this re:Post Knowledge Center Article, this would definitely help you to pass this error. Comment here how it goes, happy to help further.

0

For me it turned out to be a network issue during installation (AWS services were reachable). I deployed the Glue Job into the wrong Subnet and I was therefore not able to connect to the NAT Gateways (Internet) and install pip modules (same would apply to your Security Group settings for the Job). Maybe this helps someone else too

etwillbefine
con risposta 9 mesi fa
0

Thanks for the response. yes my connection was not able to reach out to internet. I modified the NAT gateway and it worked

Vivek
con risposta 3 mesi fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande

Contenuto pertinente