Is it possible to run a JDK11 application on EMR serverless?

0

Is it possible to run a JDK11 application on EMR serverless? I tried and got an error:

Job execution failed, please check complete logs in configured logging destination. ExitCode: 1. Last few exceptions: Exception in thread "main" java.lang.UnsupportedClassVersionError: [......] has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0...

Thanks!

feita há 2 anos2,7 mil visualizações
4 Respostas
2
Resposta aceita

Short answer

Not at this time.

Long answer

As outlined in the Hadoop Java Versions documentation[1], Apache Hadoop 3.3 and upper supports Java 8 and Java 11 (runtime only) and compiling Hadoop with Java 11 is not supported. Currently, EMR Serverless supports only EMR release 6.6.0 with Hadoop 3.2.1, which unfortunately supports Java 8 only. Therefore, updating the java version on EMR might lead to unintended behaviour. There is a feature request to add Java 11 support in EMR. EMR updates are listed in link[2].

For verification purposes, I run the following simple Spark job in EMR Serverless to check the available Java versions:

import os

if __name__ == "__main__":
    print(os.listdir("/usr/lib/jvm"))

The job returned the following: ['jre', 'java-1.8.0-amazon-corretto.x86_64', 'java']. This means that the only version available in EMR Serverless for now is Java 8.

Documentation links:

respondido há 2 anos
profile picture
ESPECIALISTA
avaliado há 4 meses
0

I followed the steps provided in documentation to create custom image of JDK 11, however, I am still unable to run my job. I had set:

--conf spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.16.0.8-1.amzn2.0.1.x86_64 
--conf spark.driverEnv.JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.16.0.8-1.amzn2.0.1.x86_64

I have posted my detailed query here: https://repost.aws/en/questions/QU-LEd1uM2Rr2p82ZNOvv9xg/emr-serverless-custom-image-not-running Has anyone of you tried the scenario with latest emr serverless release?

Regards Tapan

respondido há um ano
0

I am able to run that. I had not added emr-serverless in the driver env variable. It should have been spark.emr-serverless.driverEnv.JAVA_HOME. Thanks guys.

respondido há um ano

Você não está conectado. Fazer login para postar uma resposta.

Uma boa resposta responde claramente à pergunta, dá feedback construtivo e incentiva o crescimento profissional de quem perguntou.

Diretrizes para responder a perguntas