AWS Glue job/notebook access to Spark package (Apache Sedona, etc.)

Thank you for this helpful info., Gonzalo.

As far as Glue interactive notebook "magics," is it a bug that %additional_python_modules only allows for a single package and not multiple that are separated by commas. Also in the notebook, it does not allow for installing a package and specifying a custom index URL (even wrapped in double quotes) because it complains about the space.

And in a Glue job, the --extra-jars param only allows for 256 characters, which is only enough for one or two jar s3 paths.

Also, --extra-jars key and value disappear after saving the Glue job (in the console) and running the job. When I go to try again, the param is gone.

The python modules parameter expects modules (an optionally versions), I believe there was another parameter when using jobs to specify a custom index but don't remember. The reason --extra-jars disappears in the job is that it has it's own configuration box and it's moved there. Maybe the magic has a length limitation, I've seen people adds lots of jars directly in jobs. Most plugins offer a "uberjar" version for convenience or you can build one yourself.

Gonzalo, I was able to get this to work in my notebook using the following:

• %spark_conf (only allows for a single value and no comma-separated)
• %extra_jars (comma separated s3 paths directly to two jars; .zip did not work)

However, the same notebook being run as a job fails due to not finding the sedona module. Is there a bug perhaps in. AWS Glue jobs. If I use the equivalent params in a separate test job, it also fails to find the spark jars.

• --spark-conf
• --extra-jars

--extra-py-files totally worked for the Sedona-related .jar files I downloaded. Thank you so much!

The primary things left to figure out are

• custom spark conf settings (e.g., spark.serializer=org.apache.spark.serializer.KryoSerializer, spark.logConf=true", spark.sql.sources.partitionOverwriteMode=dynamic, etc.)
• installing sedona works via --additional-python-modules but what about private packages with a custom index URL?

My preferred option to set that is using SparkSession.builder (since the serializer cannot be changed once the session is create) but you can also use --conf or the magic if you want. I think you can do custom index using --python-modules-installer-option (see further info in the docs), remember you can always download the whl to s3 and reference it there.

When requiring multiple spark config settings in a Glue job, the following works. However, I am not aware of how to specify multiple spark config settings in a Glue notebook. In a Glue notebook, %spark_conf works but only for a single config setting.

    spark_context = SparkContext.getOrCreate()
    glue_context = GlueContext(spark_context)

    spark_session = (
        glue_context.spark_session.builder.appName(<session_name>)
        .config("spark.logConf", "true")
        .config("spark.sql.sources.partitionOverwriteMode", "dynamic")
        .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
        .config(
            "spark.kryo.registrator",
            "org.apache.sedona.core.serde.SedonaKryoRegistrator",
        )
        .enableHiveSupport()
        .getOrCreate()
    )