Is it possible to run a JDK11 application on EMR serverless?

0

Is it possible to run a JDK11 application on EMR serverless? I tried and got an error:

Job execution failed, please check complete logs in configured logging destination. ExitCode: 1. Last few exceptions: Exception in thread "main" java.lang.UnsupportedClassVersionError: [......] has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0...

Thanks!

preguntada hace 2 años2,7 mil visualizaciones
4 Respuestas
2
Respuesta aceptada

Short answer

Not at this time.

Long answer

As outlined in the Hadoop Java Versions documentation[1], Apache Hadoop 3.3 and upper supports Java 8 and Java 11 (runtime only) and compiling Hadoop with Java 11 is not supported. Currently, EMR Serverless supports only EMR release 6.6.0 with Hadoop 3.2.1, which unfortunately supports Java 8 only. Therefore, updating the java version on EMR might lead to unintended behaviour. There is a feature request to add Java 11 support in EMR. EMR updates are listed in link[2].

For verification purposes, I run the following simple Spark job in EMR Serverless to check the available Java versions:

import os

if __name__ == "__main__":
    print(os.listdir("/usr/lib/jvm"))

The job returned the following: ['jre', 'java-1.8.0-amazon-corretto.x86_64', 'java']. This means that the only version available in EMR Serverless for now is Java 8.

Documentation links:

respondido hace 2 años
profile picture
EXPERTO
revisado hace 4 meses
0

I followed the steps provided in documentation to create custom image of JDK 11, however, I am still unable to run my job. I had set:

--conf spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.16.0.8-1.amzn2.0.1.x86_64 
--conf spark.driverEnv.JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.16.0.8-1.amzn2.0.1.x86_64

I have posted my detailed query here: https://repost.aws/en/questions/QU-LEd1uM2Rr2p82ZNOvv9xg/emr-serverless-custom-image-not-running Has anyone of you tried the scenario with latest emr serverless release?

Regards Tapan

respondido hace un año
0

I am able to run that. I had not added emr-serverless in the driver env variable. It should have been spark.emr-serverless.driverEnv.JAVA_HOME. Thanks guys.

respondido hace un año

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas