Launch Spark in Full Table Access (FTA) mode in EMR on EC2

Overview

Spark in Amazon EMR on EC2, EMR Serverless, EMR on EKS and AWS Glue 5.0+ supports two modes of cluster integration to work with AWS Lake Formation managed Data Catalog tables - Full Table Access (FTA) mode and Fine-Grained Access Control (FGAC) mode. Reference for FTA vs FGAC article.

The key steps are explained in Lake Formation full table access for Amazon EMR on EC2. I am sharing the important highlights you need to focus on for EMR Spark FTA.

Reference document for the EMR security configuration and runtime role permissions are the same as that of EMR FGAC setup.

For an EMR cluster to work with Lake Formation managed tables, there are two key things to note. The role you want to use for managing access to your Data Catalog tables and the EMR security configuration. You have a choice here.

(1) EMR EC2 Instance Profile role:

If you like to manage access to your Data Catalog tables with the EC2 instance profile role, you can use a cluster without security configuration. In this use-case, the instance profile role is at the cluster level. Hence, anyone with access to the cluster will get access to all the catalog tables that the instance profile role can access.

(2) EMR run time role:

If you like to maintain access to the Data Catalog tables with specific roles for jobs and in a use case where multiple roles/teams are accessing the same cluster, use EMR runtime role for each step. To use an EMR run time role, setup a security configuration with runtime role. In this case, Lake Formation permissions will be evaluated against the run time role and permissions are managed and segregated at step level.

This EMR runtime role should not have direct S3 permissions in its IAM policy to access the data lake locations registered with Lake Formation. The IAM action lakeformation:GetDataAccess in its IAM policy allows the role to go through Lake Formation permissions, to access Lake Formation managed tables and their S3 data locations.

A sample CLI command for creating the required EMR Security configuration is shown below. In-transit encryption is required. You can use your choice of certificate provider. EMR added support for an EMR-managed certificate provider for in-transit encryption. I am showing that in my example below. More details in the section Specifying encryption options using the console.

aws emr create-security-configuration --name "lf-fta-runtimerole-emr-managed-cert" --security-configuration '{
    "AuthorizationConfiguration": {
      "IAMConfiguration": {
        "EnableApplicationScopedIAMRole": true
      }
    },
    "EncryptionConfiguration": {
      "EnableInTransitEncryption": true,
      "EnableAtRestEncryption": false,
      "InTransitEncryptionConfiguration": {
        "TLSCertificateConfiguration": {
          "CertificateProviderType": "EMR"
        }
      }
    }
}'

Also, in the Lake Formation console, under Application integration settings, you need to allow external engines to access data in the current account's S3 locations with full table access. Screenshot is shown below from Lake Formation console.

All other launch steps for EMR cluster are the same as standard cluster launch steps. You can use the Region's default VPC and public subnet, and the EC2 security groups created by EMR, to get started with a simple test cluster.

Few additional call outs for cluster launch:

1. For an Iceberg table read and write, launch a cluster with iceberg-default configuration pointing to iceberg jar. Reference doc Use an Iceberg cluster with Spark.
2. Enable GDC integration for Apache Hive and Apache Spark metastore.
3. Sample Spark config to use in your Spark job to work with Iceberg tables is shown below. "spark.sql.catalog.spark_catalog.glue.lakeformation-enabled" = "true" is the important property that enables the Spark job in FTA mode to access Lake Formation managed tables.

spark = (SparkSession
            .builder
            .appName("SparkSQL_read_iceberg_table")
            .config(f"spark.sql.catalogImplementation", "hive")
            .config(f"spark.hive.metastore.client.factory.class", "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory")
            .config(f"spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkCatalog")
            .config(f"spark.sql.catalog.spark_catalog.type", "glue")
            .config(f"spark.sql.defaultCatalog", "spark_catalog")
            .config(f"spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
            .config(f"spark.sql.catalog.spark_catalog.warehouse", "s3://your-bucket/your-prefix-for-iceberg-table/")
            .config(f"spark.sql.catalog.spark_catalog.client.region", "us-east-1")
            .config(f"spark.sql.catalog.spark_catalog.glue.account-id", "123456789012")
            .config(f"spark.sql.catalog.spark_catalog.glue.lakeformation-enabled", "true")
            .config(f"spark.sql.catalog.dropDirectoryBeforeTable.enabled", "true")
            .getOrCreate())

You can combine the points 1, 2 and 3 mentioned above in a single JSON configuration file and use it to launch the EMR cluster. A sample configurations.json with Lake Formation Full Table Access (FTA) for Iceberg tables on EMR 7.12.0 is provided below.

[
 {
   "Classification": "iceberg-defaults",
   "Properties": {
     "iceberg.enabled": "true"
   }
 },
 {
   "Classification": "spark-hive-site",
   "Properties": {
     "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
   }
 },
 {
   "Classification": "spark-defaults",
   "Properties": {
     "spark.sql.catalogImplementation": "hive",
     "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
     "spark.sql.catalog.spark_catalog": "org.apache.iceberg.spark.SparkSessionCatalog",
     "spark.sql.catalog.spark_catalog.type": "glue",
     "spark.sql.catalog.spark_catalog.warehouse": "s3://your-bucket/your-prefix-for-iceberg-table/",
     "spark.sql.catalog.spark_catalog.client.region": "us-east-1",
     "spark.sql.catalog.spark_catalog.glue.account-id": "123456789012",
     "spark.sql.catalog.spark_catalog.glue.lakeformation-enabled": "true",
     "spark.sql.catalog.dropDirectoryBeforeTable.enabled": "true"
   }
 }
]

Additional Resources

1. An earlier blog on this topic