Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

AWS Glue 5.0 --additional-python-modules no longer works as it did in Glue 4.0

0

I see there is a change to now support requirements.txt files, however we still prefer to load our internal wheel files via S3 buckets. In Glue 4.0 we were doing this via a key of '--additional-python-modules' with the value 's3://<redacted>/commons-0-py3-none-any.whl,s3://<redacted>/more-packages.whl,pymongo'

When we try running the same thing in Glue 5.0 the job fails out the gate due to:

24/12/06 18:33:55 ERROR GlueBootstrap: Bootstrapping failed 
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: One or more modules are failed to install. Please refer the logs for details.
	at com.amazonaws.services.glue.ExecutorServiceUtils.lambda$invokeTasksResolveFutures$1(ExecutorServiceUtils.java:32) ~[EMRGlueBootstrap-5.0-jar-with-dependencies.jar:5.0]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
	at com.amazonaws.services.glue.ExecutorServiceUtils.invokeTasksResolveFutures(ExecutorServiceUtils.java:34) ~[EMRGlueBootstrap-5.0-jar-with-dependencies.jar:5.0]
	at com.amazonaws.services.glue.GlueBootstrap.setUpGlueAndUserLibs(GlueBootstrap.java:136) ~[EMRGlueBootstrap-5.0-jar-with-dependencies.jar:5.0]
	at com.amazonaws.services.glue.GlueBootstrap.bootstrap(GlueBootstrap.java:78) ~[EMRGlueBootstrap-5.0-jar-with-dependencies.jar:5.0]
	at com.amazonaws.services.glue.GlueBootstrap.main(GlueBootstrap.java:64) [EMRGlueBootstrap-5.0-jar-with-dependencies.jar:5.0]
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: One or more modules are failed to install. Please refer the logs for details.
	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
	at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
	at com.amazonaws.services.glue.ExecutorServiceUtils.lambda$invokeTasksResolveFutures$1(ExecutorServiceUtils.java:30) ~[EMRGlueBootstrap-5.0-jar-with-dependencies.jar:5.0]

Looking at the release notes I don't see anything mentioning that this method is no longer supported in Glue 5.0?

Topics
Analytics
Tags
AWS Glue
Language
English

asked 2 years ago2.8K views

2 Answers
0

You're correct that AWS Glue 5.0 has introduced support for requirements.txt files, but this doesn't mean that the previous method of using --additional-python-modules is no longer supported. In fact, AWS Glue 5.0 should still support the method you were using in Glue 4.0 for loading internal wheel files via S3 buckets.

The error message you're seeing suggests that there might be an issue with installing one or more of the modules you're trying to add. This could be due to compatibility issues with the new Python version (3.11) in Glue 5.0, or perhaps there's a problem with accessing the S3 buckets where your wheel files are stored.

To troubleshoot this issue:

Double-check that the S3 paths to your wheel files are correct and accessible.
Ensure that your wheel files are compatible with Python 3.11, which is the version used in Glue 5.0.
Try installing the modules one at a time to identify which one might be causing the issue.
Review the full logs for more detailed error messages about the module installation failures.

If you continue to face issues, you might want to consider using the new requirements.txt method, which allows you to specify your dependencies in a file:

Create a requirements.txt file with your dependencies, including the S3 paths to your wheel files.
Upload this file to an S3 bucket.
Modify your job parameters to include:
    Key: --python-modules-installer-option Value: -r
    Key: --additional-python-modules Value: s3://path_to_your_requirements.txt

This new method should allow you to continue using your internal wheel files while taking advantage of the improvements in Glue 5.0.

If the problem persists, it would be advisable to contact AWS support for further assistance, as there might be an underlying issue specific to your setup or the new Glue 5.0 environment. Sources Using Python libraries with AWS Glue - AWS Glue Introducing AWS Glue 5.0 for Apache Spark | AWS Big Data Blog Introducing AWS Glue 5.0 - AWS

AWS

answered 2 years ago

  • rePost-User-0640902
    2 years ago

    I'm fairly certain there is just a bug in the Glue 5.0 setup at this point. I have added a requirements.txt file to my S3 bucket, the Glue role the is executing the job has full access to the S3 bucket (*), yet the job cannot even find the requirements.txt file, let alone attempt to download anything from it.

    24/12/12 14:24:43 INFO PythonModuleInstaller: ERROR: Invalid requirement: 's3://<redacted>/glue/requirements.txt'Hint: It looks like a path. File 's3://<redacted>/glue/requirements.txt' does not exist.

    This is using your recommendations from above

    --python-modules-installer-option | -r --additional-python-modules | s3://<redacted>/glue/requirements.txt

-2

You're correct that AWS Glue 5.0 has introduced support for requirements.txt files, but this doesn't mean that the previous method of using --additional-python-modules is no longer supported. In fact, AWS Glue 5.0 should still support the method you were using in Glue 4.0 for loading internal wheel files via S3 buckets.

The error message you're seeing suggests that there might be an issue with installing one or more of the modules you're trying to add. This could be due to compatibility issues with the new Python version (3.11) in Glue 5.0, or perhaps there's a problem with accessing the S3 buckets where your wheel files are stored.

To troubleshoot this issue:

  1. Double-check that the S3 paths to your wheel files are correct and accessible.
  2. Ensure that your wheel files are compatible with Python 3.11, which is the version used in Glue 5.0.
  3. Try installing the modules one at a time to identify which one might be causing the issue.
  4. Review the full logs for more detailed error messages about the module installation failures.

If you continue to face issues, you might want to consider using the new requirements.txt method, which allows you to specify your dependencies in a file:

  1. Create a requirements.txt file with your dependencies, including the S3 paths to your wheel files.
  2. Upload this file to an S3 bucket.
  3. Modify your job parameters to include:
    • Key: --python-modules-installer-option Value: -r
    • Key: --additional-python-modules Value: s3://path_to_your_requirements.txt

This new method should allow you to continue using your internal wheel files while taking advantage of the improvements in Glue 5.0.

If the problem persists, it would be advisable to contact AWS support for further assistance, as there might be an underlying issue specific to your setup or the new Glue 5.0 environment.
Sources
Using Python libraries with AWS Glue - AWS Glue
Introducing AWS Glue 5.0 for Apache Spark | AWS Big Data Blog
Introducing AWS Glue 5.0 - AWS

answered 2 years ago

  • Michel
    2 years ago

    I am having the same problem. With version 4.0, the modules are installed correctly. I have already tried using the suggested configurations: --python-modules-installer-option: -r --additional-python-modules: s3://path_to_your_requirements.txt

    However, the requirements.txt file is not found. My Glue role is correctly set with full access to the bucket.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content