LAUNCH ERROR | Python Module Installer indicates modules that failed to install, check logs from the PythonModuleInstaller.Please refer logs for details.
I am using python module in aws glue
But Glue job fails in randomly
Error is
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them. pymssql from https://files.pythonhosted.org/packages/9a/78/6d78d3ed551341d86cc3b56dc944d0c6dea5ceab7c634b9b6e8fc51242f2/pymssql-2.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl: Expected sha256 06883bc9bdb297ae9132d9371b5b1a3a223c8f93dd6a87d1c112c6a688f26d53 Got 11669a57c78a2d6e9eb6eb561ada447578f0b3d3caecb514a262d756a337a734[notice] A new release of pip is available: 23.0.1 -> 25.0.1[notice] To update, run: pip3 install --upgrade pip
- Newest
- Most votes
- Most comments
The error you're encountering suggests that there's an issue with installing Python modules in your AWS Glue job. This can happen for a few reasons:
-
Module compatibility: The modules you're trying to install may not be compatible with the Python version used by your Glue job. Make sure the modules you're specifying are compatible with the Python version in your Glue environment.
-
Memory issues: Sometimes, module installation failures can occur due to insufficient memory. If your job is running out of memory during the module installation process, it could lead to random failures.
-
Incorrect module specification: Ensure that you've correctly specified the module names and versions in the --additional-python-modules parameter. The format should be module_name==version_number, with multiple modules separated by commas.
-
Network issues: If the job is having trouble accessing the Python Package Index (PyPI) to download the modules, it could result in installation failures.
To troubleshoot this issue:
-
Check the logs from the PythonModuleInstaller for more detailed information about which modules failed to install and why.
-
Verify that all module names and versions are correctly specified in the --additional-python-modules parameter.
-
Try increasing the worker type or number of workers for your Glue job to provide more memory for the module installation process.
-
If possible, test the installation of these modules in a local environment that mirrors your Glue environment to ensure they're compatible.
-
Consider creating a custom runtime environment that includes all necessary packages, or package your required modules in a ZIP file and upload it to an S3 bucket, then specify this in your job's Python library path setting.
If the issue persists, you may need to review the specific error messages in the PythonModuleInstaller logs to identify any particular modules causing problems and address them individually.
Sources
AWS Glue Visual ETL - Install/ import packages | AWS re:Post
answered a year ago
Relevant content
asked 3 years ago
