Is it possible to run a JDK11 application on EMR serverless?

Question

Is it possible to run a JDK11 application on EMR serverless? I tried and got an error:
```
Job execution failed, please check complete logs in configured logging destination. ExitCode: 1. Last few exceptions: Exception in thread "main" java.lang.UnsupportedClassVersionError: [......] has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0...
```

Thanks!

Accepted Answer

### Short answer
Not at this time.

### Long answer
As outlined in the Hadoop Java Versions documentation[1], Apache Hadoop 3.3 and upper supports Java 8 and Java 11 (runtime only) and compiling Hadoop with Java 11 is not supported. 
Currently, EMR Serverless supports only EMR release 6.6.0 with Hadoop 3.2.1, which unfortunately supports Java 8 only. Therefore, updating the java version on EMR might lead to unintended behaviour.
There is a feature request to add Java 11 support in EMR.   EMR updates are listed in link[2].

For verification purposes, I run the following simple Spark job in EMR Serverless to check the available Java versions:
```python
import os

if __name__ == "__main__":
    print(os.listdir("/usr/lib/jvm"))
```

The job returned the following: ['jre', 'java-1.8.0-amazon-corretto.x86_64', 'java']. This means that the only version available in EMR Serverless for now is Java 8.

### Documentation links:

* [1] https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions 
* [2] https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-whatsnew.html

Answer

I followed the steps provided in documentation to create custom image of JDK 11, however, I am still unable to run my job.
I had set:
```
--conf spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.16.0.8-1.amzn2.0.1.x86_64 
--conf spark.driverEnv.JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.16.0.8-1.amzn2.0.1.x86_64
```
I have posted my detailed query here: [https://repost.aws/en/questions/QU-LEd1uM2Rr2p82ZNOvv9xg/emr-serverless-custom-image-not-running]()
Has anyone of you tried the scenario with latest emr serverless release?

Regards
Tapan

Answer

I am able to run that. I had not added `emr-serverless` in the driver env variable. It should have been `spark.emr-serverless.driverEnv.JAVA_HOME`.
Thanks guys.

Answer

Updated: now it possible, you can use emr-serverless 6.9.0 and custom image with java 11

Documentation links:

[1] https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/application-custom-image.html

[2] https://aws.amazon.com/ru/blogs/big-data/add-your-own-libraries-and-application-dependencies-to-spark-and-hive-on-amazon-emr-serverless-with-custom-images/

Is it possible to run a JDK11 application on EMR serverless?

Short answer

Long answer

Documentation links:

관련 콘텐츠