how to enable delta lake in EMR serverless.

Hello Muthukumar,

--packages not required if you are using EMR 6.9.0 onwards as the delta lake jar will be shipped to EMR image by default. In older versions, you have to import the compatible OSS Deltalake dependencies to make the session worked as expected as in your the dependent version might not be compatible. Certainly, I tried the latest version in EMR-S with below steps tested for your reference, working fine.

1. Configure your Spark session.

Configure the Spark Session. Set up Spark SQL extensions to use Delta lake.

%%configure -f
{
    "conf": {
        "spark.sql.extensions" : "io.delta.sql.DeltaSparkSessionExtension",
        "spark.sql.catalog.spark_catalog": "org.apache.spark.sql.delta.catalog.DeltaCatalog",
        "spark.jars": "/usr/share/aws/delta/lib/delta-core.jar,/usr/share/aws/delta/lib/delta-storage.jar,/usr/share/aws/delta/lib/delta-storage-s3-dynamodb.jar",
        "spark.hadoop.hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
    }
}

2. Create a Delta lake Table

We will create a Spark Dataframe with sample data and write this into a Delta lake table.

tableName = "delta_table"
basePath = "s3://<Your S3 bucket>/test/delta/" + tableName

data = spark.createDataFrame([
 ("100", "2015-01-01", "2015-01-01T13:51:39.340396Z"),
 ("101", "2015-01-01", "2015-01-01T12:14:58.597216Z"),
 ("102", "2015-01-01", "2015-01-01T13:51:40.417052Z"),
 ("103", "2015-01-01", "2015-01-01T13:51:40.519832Z")
],["id", "creation_date", "last_update_time"])

data.write.format("delta"). \
  save(basePath)

3. Query the table

We will read the table using spark.read into a Spark dataframe

df = spark.read.format("delta").load(basePath)
df.show()

+---+-------------+--------------------+
| id|creation_date|    last_update_time|
+---+-------------+--------------------+
|102|   2015-01-01|2015-01-01T13:51:...|
|103|   2015-01-01|2015-01-01T13:51:...|
|101|   2015-01-01|2015-01-01T12:14:...|
|100|   2015-01-01|2015-01-01T13:51:...|
+---+-------------+--------------------+