- 最新
- 投票最多
- 评论最多
Short answer
Not at this time.
Long answer
As outlined in the Hadoop Java Versions documentation[1], Apache Hadoop 3.3 and upper supports Java 8 and Java 11 (runtime only) and compiling Hadoop with Java 11 is not supported. Currently, EMR Serverless supports only EMR release 6.6.0 with Hadoop 3.2.1, which unfortunately supports Java 8 only. Therefore, updating the java version on EMR might lead to unintended behaviour. There is a feature request to add Java 11 support in EMR. EMR updates are listed in link[2].
For verification purposes, I run the following simple Spark job in EMR Serverless to check the available Java versions:
import os if __name__ == "__main__": print(os.listdir("/usr/lib/jvm"))
The job returned the following: ['jre', 'java-1.8.0-amazon-corretto.x86_64', 'java']. This means that the only version available in EMR Serverless for now is Java 8.
Documentation links:
Updated: now it possible, you can use emr-serverless 6.9.0 and custom image with java 11
Documentation links:
[1] https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/application-custom-image.html
I followed the steps provided in documentation to create custom image of JDK 11, however, I am still unable to run my job. I had set:
--conf spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.16.0.8-1.amzn2.0.1.x86_64
--conf spark.driverEnv.JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.16.0.8-1.amzn2.0.1.x86_64
I have posted my detailed query here: https://repost.aws/en/questions/QU-LEd1uM2Rr2p82ZNOvv9xg/emr-serverless-custom-image-not-running Has anyone of you tried the scenario with latest emr serverless release?
Regards Tapan
I am able to run that. I had not added emr-serverless
in the driver env variable. It should have been spark.emr-serverless.driverEnv.JAVA_HOME
.
Thanks guys.
相关内容
- AWS 官方已更新 1 个月前
- AWS 官方已更新 1 个月前