AWS Glue Visual ETL - Install/ import packages

Installing and using custom Python packages in AWS Glue Visual ETL jobs can be achieved, but it requires a slightly different approach than using !pip install directly in the Custom Transform node.

To use a custom Python package like 'textdistance' in your AWS Glue Visual ETL job, you have a few options:

1. Use the --additional-python-modules parameter: When creating or editing your Glue job, you can specify additional Python modules to be installed. In the job properties or advanced properties section, add the parameter --additional-python-modules and set its value to the package you want to install, for example: textdistance==4.2.1 (specify the version you need).

2. Include the package in a ZIP file: If the package is not available through pip or you need a specific version, you can create a ZIP file containing the package and its dependencies. Upload this ZIP file to an S3 bucket, and then specify the S3 path in your job's Python library path setting.

3. Use a custom runtime environment: Create a custom runtime environment that includes the packages you need, and use this environment for your Glue job.

After setting up one of these methods, you should be able to import and use the package in your Custom Transform node without needing to use !pip install.

For example, in your Custom Transform code, you could then simply use:

import textdistance

# Your transformation logic using textdistance

If you prefer to stick with Visual ETL and avoid creating a separate notebook, the first two options are your best bet. They allow you to add the necessary packages while still using the visual interface for job creation and management.

Remember that when using custom packages, you need to ensure that they are compatible with the Python version and environment used by your AWS Glue job. Always test your job thoroughly after adding custom packages to ensure everything works as expected.

Sources
Using Python libraries with AWS Glue - AWS Glue
Building visual ETL jobs with AWS Glue Studio - AWS Glue
AWS Glue concepts - AWS Glue