Cross-Account Access for AWS Glue Job (Account A) to Access Glue Data Catalog and Iceberg Tables (Account B)

Hi All,

I'm trying to enable cross-account access where a Glue job running in Account A needs to read/write Iceberg tables that reside in the Glue Data Catalog of Account B.

What I’ve Tried: Approach 1: Direct IAM Role Access (Account A role) + Resource Policy in Account B

1. Glue job in Account A runs with an IAM role (let’s call it GlueRoleA)
2. In Account B, Resource policy on the Glue Data Catalog grants access to GlueRoleA
3. S3 bucket policy allows access to data bucket from GlueRoleA
4. LF Data permission grants (including ALL and SUPER) are given to GlueRoleA

Approach 2: Assume Role

1. In Account A, Glue job role (GlueRoleA) assumes a role in Account B (GlueRoleB)
2. In Account B, GlueRoleB has all required Glue and Lake Formation permissions (including ALL and SUPER)
3. S3 bucket policy allows access from GlueRoleB
4. Lake Formation grants are provided to GlueRoleB

In both approaches, when using GlueContext to access Iceberg tables, I encounter the following error: TABLE_OR_VIEW_NOT_FOUND

Please find my code below:

import sys
import boto3
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from pyspark.sql import SparkSession
# Initialize Spark Context with the required Spark configurations for Iceberg
spark = SparkSession.builder \
    .appName("CrossAccountGlueIcebergJob") \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config("spark.sql.catalog.glue_catalog", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.glue_catalog.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog") \
    .config("spark.sql.catalog.glue_catalog.warehouse", "s3://<bucket_name>/iceberg_warehouse/") \
    .config("spark.sql.catalog.glue_catalog.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") \
    .config("spark.sql.catalog.glue_catalog.catalog-id", "111122223333") \
    .config(f"spark.sql.catalog.glue_catalog.glue.account-id", "111122223333") \
    .config("spark.sql.defaultCatalog", "glue_catalog") \
    .getOrCreate()

# Initialize GlueContext
glueContext = GlueContext(spark.sparkContext)
# Debugging: Print Spark configurations related to Iceberg catalog setup
print("Spark Configurations for Iceberg Catalog:")
for key, value in spark.sparkContext.getConf().getAll():
    if "spark.sql.catalog" in key:
        print(f"{key} = {value}")
# Create a Boto3 Glue Client for cross-account operations
glue_client = boto3.client('glue', region_name='us-east-1')
account_b_id = "111122223333"  # Replace with Account B ID
# List tables in the specific database in Account B
print("Fetching tables from Account B Glue Data Catalog...")
response = glue_client.get_tables(CatalogId=account_b_id, DatabaseName='welcome_db')
print("Tables in database 'welcome_db':")
for tbl in response['TableList']:
    print("Table:", tbl['Name'])

print("Attempting to read table")
 
df_csv = glueContext.create_data_frame.from_catalog(
    database="welcome_db",
    table_name="dummy",
    catalog_id = account_b_id
)
 
df_csv.show(10)
 
 
for key, value in spark.sparkContext.getConf().getAll():
    if "spark.sql.catalog" in key:
        print(f"{key} = {value}")
 
try:
    df_iceberg = glueContext.create_data_frame.from_catalog(
        database="welcome_db",
        table_name="sample_iceberg_tbl",
        catalog_id = account_b_id
    )
    df_iceberg.show(10)
except Exception as e:
    raise e
finally:
    for key, value in spark.sparkContext.getConf().getAll():
        if "spark.sql.catalog" in key:
            print(f"{key} = {value}")
 

print("Job Completed Successfully.")

Error: Error Category: QUERY_ERROR; Failed Line Number: 66; Spark Error Class: TABLE_OR_VIEW_NOT_FOUND; AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view glue_catalog.welcome_db.sample_iceberg_tbl cannot be found. Verify the spelling and correctness of the schema and catalog.

Note: df_csv table with csv data file works. But df_iceberg is not failing, TABLE_OR_VIEW_NOT_FOUND!

Appreciating your help, Thanks in Advance