AWS Glue on docker - java.lang.UnsatisfiedLinkError: no AWSGlueSimdNative in java.library.path

Question

Hi,
I'm running my AWS Glue pyspark job on docker locally based on this description - https://aws.amazon.com/blogs/big-data/develop-and-test-aws-glue-version-3-0-jobs-locally-using-a-docker-container.
I'm getting such exception when I try to display data from glue dynamic frame: gdf.show(5)

23/02/06 11:20:09 WARN SimdCsvParser$: Could not find AWSGlueSimdNative lib from /home/glue_user/native_libs/amd64/hadoop-lzo-native:/home/glue_user/native_libs/amd64/hadoop-native/:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
java.lang.UnsatisfiedLinkError: no AWSGlueSimdNative in java.library.path
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
        at java.lang.Runtime.loadLibrary0(Runtime.java:843)
        at java.lang.System.loadLibrary(System.java:1136)
        at com.amazonaws.services.glue.readers.csv.SimdCsvParser$.loadNativeLib(SimdCsvParser.scala:366)
        at com.amazonaws.services.glue.readers.csv.SimdCsvParser$.isAllowed(SimdCsvParser.scala:361)
        at com.amazonaws.services.glue.readers.CSVReader.init(CSVReader.scala:258)
        at com.amazonaws.services.glue.hadoop.TapeHadoopColumnarReader.initialize(TapeHadoopRecordReader.scala:272)

Do you know if some native library is missing for image amazon/aws-glue-libs:glue_libs_4.0.0_image_01 ?
Regards,

Subramanya V · Answer

Hello @rePost-User-8465222,

The `WARN` message outlined in your query is occurring because the native binaries for `vectorized SIMD CSV reader` are not available in Local development environment (including docker images). As of now, `optimizePerformance` flag can be set only in AWS Glue ETL Job system.

We are in the process of updating our documentation to outline this limitation. To suppress the `WARN` message, you can remove the `optimizePerformance` flag or set it to `False`.

AWS Glue on docker - java.lang.UnsatisfiedLinkError: no AWSGlueSimdNative in java.library.path

回答を追加する

関連するコンテンツ