- Newest
- Most votes
- Most comments
Hi ,
I think there is a typo in the parameter you are using to install the additional package, it should be:
--additional-python-modules
note the 'S' at the end.
I tested with both Glue 2.0 and 3.0 and it worked fine in my environment.
Also, when testing an additional library, you can check your error logs in Cloudwatch and look what is happening during the installation of your packages. you should be able to see an entry like:
INFO 2022-03-20 15:34:53,423 3488 com.amazonaws.services.glue.PythonModuleInstaller [main] pip3 install --user sshtunnel==0.4.0 redshift-connector
Finally, there is no need to pre-create the wheel , sshtunnel can be installed with passing the following parameter.
--additional-python-modules and as value sshtunnel==0.4.0
hope this helps
Relevant content
- asked a year ago
- asked 2 years ago
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
Thanks for sharing.
if we are not calling the wheel then glue try to install every time jobs runs ??
One more thing!
Can i use the sshtunnel forwarding in glue script.If yes please let me know.We are are trying but its not working
Code something like below
with SSHTunnelForwarder( ssh_address_or_host = (HOST, 22), ssh_username = UserID, ssh_pkey = pkey.pem, remote_bind_address=(IP, 5432), local_bind_address=('', 45432) ) as server: connection_string = 'postgresql://UID:PWD@localhost:45432/DB' engine = create_engine(connection_string)
How i can use the sshtunnel forwarding
@AWS-User-3534774
If you wan to use the wheel you can the name of the parameters need to be fixed in any case. Yes passing the parameter with the name of the library will install it every time, it should not make a big difference though.
As for the other question, could you please explain some additional details?
What are you trying to achieve?
What is the error?
Are you running the glue job in a VPC?
is the server you want to reach open to the internet or in a private subnet?
This might be worth a separate question. and if the answer was correct please accept it, it helps others.