Glue Notebook error for Delta

Hello,

You are encountering a "java.lang.ClassNotFoundException: delta.DefaultSource" due to missing Delta Lake packages in your AWS Glue notebook. To resolve the issue, please include the following configurations to ensure that the necessary Delta Lake packages are included:

%%configure
{
    "--datalake-formats": "delta",
    "--conf": "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog --conf spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore"
}

Similarly, for glue jobs, we can achieve the same using the following steps:

1. Click on the "Job details" tab.

2. Drop down the "Advanced properties" subsection

3. Click on the "Add new parameter" button under the "Job parameters" section. Add the following:

key1: --datalake-formats

value1: delta

key2: --conf

value2: spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog --conf spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore

4. Save the job

Please find the below example for the same.

%%configure
{
    "--datalake-formats": "delta",
    "--conf": "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog --conf spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore"
}

from pyspark.sql import SparkSession
from delta.tables import DeltaTable

spark = SparkSession.builder.getOrCreate()
sdf = spark.read.format("csv").option("header", True).load("s3://<bucket-name>/src_data/")

sdf.show()

sdf.write.format("delta").save("s3://<bucket-name>/delta_tgt_data/")

If you are still facing the issue even after making the above changes then please raise a support case with us here - https://docs.aws.amazon.com/awssupport/latest/user/case-management.html#creating-a-support-case

Reference: [+] https://docs.delta.io/latest/quick-start.html

Thank you and have a nice day.