How to set spark.driver.extraLibraryPath in EMR Serverless

Question

Hello, i tried run JNI code  which uses the file as a dependency lib**MylibName**.so. Before(not in EMR Serverless) i can use it just specify 
**--conf spark.driver.extraLibraryPath=/path_to_my_lib and  --conf spark.executor.extraLibraryPath=/path_to_my_lib**
, but this options not supported in Serverless following here [documentation](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/jobs-spark.html)
i found similar answer for python, but it not suit for me 
[post](https://repost.aws/questions/QUt1lKNvBBTPuTD9U8GO-SpA/emr-serverless-6-6-0-python-swig-lib-dependency), because i use Scala

Error which i got if not specify  spark.driver.extraLibraryPath and spark.executor.extraLibraryPath=
Exception in thread "main" java.lang.UnsatisfiedLinkError: no **MylibName** in java.library.path

Question: how i can specify this dependency for my spark application?

FYI: --conf spark.driver.extraJavaOptions=\"-Djava.library.path=s3://path_to_libs\" also not work correct 
it changed java.library.path Variable but i continue got UnsatisfiedLinkError 
![Enter image description here](/media/postImages/original/IMx-0hzZJgTC-f4TvK26G-2g)

Answer

I found the solution on my own and publish it here:

--conf spark.archives=s3://path_to_file/my_libs_so.tar.gz#**my_libs** --conf spark.executor.extraJavaOptions=\"-Djava.library.path=/home/hadoop/**my_libs**\" --conf spark.driver.extraJavaOptions=\"-Djava.library.path=/home/hadoop/**my_libs**\"

Archive **my_libs_so.tar.gz** should contains all  your the necessary *.so files

How to set spark.driver.extraLibraryPath in EMR Serverless

相关内容