Import External Libraries (Glue) Without Internet

Question

Hi All,

We have a client requirement where we are running our process in glue using python shell. Since they are using VPN, we do not have internet connectivity and want some of the pypi packages to be imported in our glue job for further processing.

How do we import all those packages without internet and also include their dependencies.

Please recommend the best possible solution which can help to import all the libraries without internet connectivity.

We tried creating wheel package for each library and hosting it in S3. But it didn't work either.

Answer

Glue shell (unlike Glue ETL) doesn't allow installing packages from s3 directly, but what you can do is create a Pypi repo in s3 with the things you need and ask the shell to use it instead of the internet one.  
To do that you need: 
- Create a repo on s3 with your dependencies, for instance using https://github.com/wolever/pip2pi  
- In the --additional-python-modules parameter you can specify pip flags, so you can use -i to point to the repository index on s3

Import External Libraries (Glue) Without Internet

Relevant content