如何解決 Amazon EMR 上 Spark 中的「java.lang.ClassNotFoundException」錯誤？

簡短說明

在 Spark 中發生 java.lang.ClassNotFoundException 錯誤的原因如下：

• spark-submit 作業在類別路徑中找不到相關檔案。
• 啟動程序動作或自訂組態會覆寫類別路徑。因此，類別載入器只會選取您在組態中指定的位置中存在的 JAR 檔案。

解決方法

若要解決 java.lang.ClassNotFoundException 錯誤，請查看堆疊追蹤以找出缺少的類別名稱。接著，將包含該缺少類別的自訂 JAR 路徑新增至 Spark 類別路徑。您可以在執行中的叢集、新的叢集，或提交作業時新增自訂 JAR 路徑。

在執行中的叢集新增自訂 JAR 路徑

在 /etc/spark/conf/spark-defaults.conf 中，將您的自訂 JAR 路徑新增至錯誤堆疊追蹤中指定的類別名稱。

範例：

sudo vim /etc/spark/conf/spark-defaults.conf
spark.driver.extraClassPath <other existing jar locations>:example-custom-jar-path
spark.executor.extraClassPath <other existing jar locations>:example-custom-jar-path

注意： 將 example-custom-jar-path 替換為您的自訂 JAR 路徑。

在新的叢集新增自訂 JAR 路徑

若要將自訂 JAR 路徑新增至 etc/spark/conf/spark-defaults.conf 中現有的類別路徑，請在建立新叢集時提供組態物件。請使用 Amazon EMR 5.14.0 或更新版本來建立新叢集。

若是 Amazon EMR 5.14.0 至 Amazon EMR 5.17.0，請納入以下內容：

[  {
    "Classification": "spark-defaults",
    "Properties": {
      "spark.driver.extraClassPath": "/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/home/hadoop/extrajars/*",
      "spark.executor.extraClassPath": "/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/home/hadoop/extrajars/*"
    }
  }
]

若是 Amazon EMR 5.17.0 至 Amazon EMR 5.18.0，請將 /usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar 納為額外的 JAR 路徑：

[  {
    "Classification": "spark-defaults",
    "Properties": {
      "spark.driver.extraClassPath": "/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/home/hadoop/extrajars/*",
      "spark.executor.extraClassPath": "/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/home/hadoop/extrajars/*"
    }
  }
]

若是 Amazon EMR 5.19.0 至 Amazon EMR 5.32.0，請按下方所示更新 JAR 路徑：

[
  {
    "Classification": "spark-defaults",
    "Properties": {
      "spark.driver.extraClassPath": "/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/home/hadoop/extrajars/*",
      "spark.executor.extraClassPath": "/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/home/hadoop/extrajars/*"
    }
  }
]

若是 Amazon EMR 5.33.0 至 Amazon EMR 5.36.0，請按下方所示更新 JAR 路徑：

[
  {
    "Classification": "spark-defaults",
    "Properties": {
      "spark.driver.extraClassPath": "/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/home/hadoop/extrajars/",
      "spark.executor.extraClassPath": "/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/home/hadoop/extrajars/"
    }
  }
]

若是 Amazon EMR 6.0.0 或更新版本，您無法使用組態來更新 JAR 路徑，因為 conf 檔案包含多個 JAR 路徑。此外，您更新的每個屬性組態長度不得超過 1024 個字元。若要將自訂 JAR 位置傳遞至 spark-defaults.conf，請新增啟動程序動作。如需詳細資訊，請參閱如何在啟動程序階段之後更新所有 Amazon EMR 節點？

若要新增啟動程序動作，請參閱新增自訂啟動程序動作，然後執行以下動作：

• 將 s3://example-bucket/Bootstraps/script_b.sh 替換為您的 Amazon Simple Storage Service (Amazon S3) 路徑。
• 將 /home/hadoop/extrajars/* 替換為您的自訂 JAR 檔案路徑。
• 確認 Amazon EMR 執行階段角色具有存取 Amazon S3 儲存貯體的必要權限。
  **注意：**當您新增啟動程序指令碼時，該指令碼會套用至叢集的 Spark 組態，而非特定作業。

變更 /etc/spark/conf/spark-defaults.conf 的指令碼範例：

#!/bin/bash
#
# This is an example of script_b.sh for changing /etc/spark/conf/spark-defaults.conf
#
while [ ! -f /etc/spark/conf/spark-defaults.conf ]
do
  sleep 1
done
#
# Now the file is available, do your work here
#
sudo sed -i '/spark.*.extraClassPath/s/$/:\/home\/hadoop\/extrajars\/\*/' /etc/spark/conf/spark-defaults.conf
exit 0
Launch the EMR cluster and add a bootstrap action similar to the following:
#!/bin/bash
pwd
aws s3 cp s3://example-bucket/Bootstraps/script_b.sh .
chmod +x script_b.sh
nohup ./script_b.sh &

**注意：**將 example-bucket 替換為您的 Amazon S3 儲存貯體。

在提交作業時新增自訂 JAR 路徑

若要在提交作業時新增自訂 JAR 路徑，請在執行 spark-submit 命令時使用 jars 選項。如需詳細資訊，請參閱 Apache Spark 網站上的使用 spark-submit 啟動應用程式。

spark-submit --deploy-mode client --class org.apache.spark.examples.SparkPi --master yarn spark-examples.jar 100 --jars example-custom-jar-path

**注意：**將 example-custom-jar-path 替換為您的自訂 JAR 路徑。為防止類別衝突，使用 jars 選項時請勿納入標準 JAR。例如，請勿納入 spark-core.jar，因為該檔案已存在於叢集內。如需詳細資訊，請參閱設定 Spark。

相關資訊

Apache Spark 網站上的 Spark 組態

如何解決 Amazon EMR 上 Spark 中的「Container killed by YARN for exceeding memory limits」錯誤？