- Newest
- Most votes
- Most comments
Hello,
I have replicated the use case in Glue 4.0 by using the scikit-learn version 1.1.3 which is default version installed for Glue 4.0 and after defining the same in glue job parameters and it works without any errors as follows:
--additional-python-modules scikit-learn==1.1.3
I have also tried to print the scikit-learn version in Glue 4.0 that returned 1.1.3 which confirms that Glue 4.0 uses default version of scikit-learn version 1.1.3.
Also checking the source code of scikit-learn 0.24.2 I can see there is no “StratifiedGroupKFold” library under the path scikit-learn-0.24.2 2/sklearn/model_selection/ init.py. You can see the source code of scikit-learn in the reference document[1].
Further running the same Glue 4.0 job by specifying the scikit-learn==0.24.2 in additional-python-modules parameter, job fails with similar error as you mentioned in your query:
ImportError: cannot import name 'StratifiedGroupKFold' from 'sklearn.model_selection' (/home/spark/.local/lib/python3.10/site-packages/sklearn/model_selection/init.py)
Also I tried adding the following parameters, and the Glue job worked without any errors:
--additional-python-modules: scikit-learn --python-modules-installer-option: --upgrade
I would suggest you to create a new job with Glue 4.0 and either specify the additional-python-modules parameter as scikit-learn==1.1.3 or simply run the job without specifying any parameter.
If the issue still persists, in order to troubleshoot further, please feel free to open a support case with AWS by specifying the error along with job run ID using the following link. We will be happy to assist you.
Reference: [1] https://github.com/scikit-learn/scikit-learn/releases/tag/0.24.2
Relevant content
- Accepted Answerasked 6 months ago
- asked 2 years ago
- asked 9 months ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 4 months ago
- How can I use a Lambda function to automatically start an AWS Glue job when a crawler run completes?AWS OFFICIALUpdated 2 years ago