Cross region Same Account Glue Job ETL - from Parquet to iceberg format

Our current setup runs the AWS Glue job in us-west-1 while our raw Parquet data is stored there. However, our target AWS Glue Data Catalog (for Iceberg table definitions) resides in us-east-2. By default, both Glue and Apache Iceberg try to interact with the local region’s catalog, which in our case means they attempt to perform operations (e.g. glue:GetTable or glue:GetDatabase) in us-west-1. This leads to authorization errors, as our IAM policy only grants permissions on the us-east-2 catalog. The Glue Database and Crawler stay on the default provider which is us-east-2

How can we can override the default region in our Spark configurations so that all metadata operations (table creation, lookups, etc.) are directed to us-east-2 instead of us-west-1??

Code:

import sys
from awsglue.utils import getResolvedOptions
from pyspark.sql import SparkSession

args = getResolvedOptions(sys.argv, ['SOURCE_BUCKET', 'DEST_BUCKET', 'JOB_NAME'])
source_bucket = args['SOURCE_BUCKET']
dest_bucket   = args['DEST_BUCKET']

spark = (SparkSession.builder
    .appName("ParquetToIceberg")
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
    .config("spark.sql.catalog.glue_catalog", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.glue_catalog.warehouse", f"s3://{dest_bucket}/iceberg-data/")
    .config("spark.sql.catalog.glue_catalog.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog")
    .config("spark.sql.catalog.glue_catalog.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
    .config("spark.sql.catalog.my_catalog.client.assume-role.region", "us-east-2")
    .getOrCreate()
)

source_path = f"s3://{source_bucket}/raw-parquet/"
df = spark.read.parquet(source_path)

table_name = "glue_catalog.iceberg_database.my_iceberg_table"
df.writeTo(table_name) \
  .tableProperty("format-version", "2") \
  .partitionedBy("username", "year", "month", "day") \
  .createOrReplace()

spark.stop()

**Minimal Terraform Setup - iam role and Glue Data Catalog **

resource "aws_iam_role" "glue_role" {
  name               = "glue-cross-region-role-${var.environment}"
  assume_role_policy = data.aws_iam_policy_document.combined_trust_policy.json
}

data "aws_iam_policy_document" "glue_policy_doc" {
  # Allow S3 read from the source bucket
  statement {
    effect = "Allow"
    actions = [
      "s3:GetObject",
      "s3:ListBucket"
    ]
    resources = [
      "arn:aws:s3:::${var.source_bucket_name}",
      "arn:aws:s3:::${var.source_bucket_name}/*"
    ]
  }

  # Allow S3 write to the destination bucket
  statement {
    effect = "Allow"
    actions = [
      "s3:PutObject",
      "s3:GetObject",
      "s3:ListBucket"
    ]
    resources = [
      aws_s3_bucket.destination_bucket.arn,
      "${aws_s3_bucket.destination_bucket.arn}/*"
    ]
  }

  # Allow CloudWatch Logs actions
  statement {
    effect    = "Allow"
    actions   = ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"]
    resources = ["*"]
  }

  # Allow Glue Data Catalog operations dynamically using the current account ID
  statement {
    effect = "Allow"
    actions = [
      "glue:GetDatabase",
      "glue:GetTable",
      "glue:CreateTable",
      "glue:UpdateTable"
    ]
    resources = [
      "arn:aws:glue:us-east-2:${data.aws_caller_identity.current.account_id}:catalog",
      "arn:aws:glue:us-east-2:${data.aws_caller_identity.current.account_id}:database/*",
      "arn:aws:glue:us-east-2:${data.aws_caller_identity.current.account_id}:table/*"
    ]
  }
}

resource "aws_iam_policy" "glue_policy" {
  name        = "glue-cross-region-policy-${var.environment}"
  description = "Policy for AWS Glue to access S3 in us-west-1 and us-east-2 and to interact with the Glue Data Catalog"
  policy      = data.aws_iam_policy_document.glue_policy_doc.json
}

resource "aws_iam_role_policy_attachment" "glue_policy_attach" {
  role       = aws_iam_role.glue_role.name
  policy_arn = aws_iam_policy.glue_policy.arn
}

############################################################
# 4. Create a Glue Database in us-east-2
############################################################

resource "aws_glue_catalog_database" "iceberg_db" {
  name = var.glue_database_name
}

############################################################
# 5. AWS Glue Job (us-east-2) reading from us-west-1
############################################################

resource "aws_glue_job" "parquet_to_iceberg" {
  provider = aws.west
  name     = "parquet-to-iceberg-job-${var.environment}"
  role_arn = aws_iam_role.glue_role.arn

  command {
    name            = "glueetl"
    script_location = "s3://${aws_s3_bucket.destination_bucket.bucket}/scripts/glue_script.py"
    python_version  = "3"
  }

  default_arguments = {
    "--job-bookmark-option" = "job-bookmark-disable"
    "--TempDir"             = "s3://${aws_s3_bucket.destination_bucket.bucket}/temp/"
    "--enable-metrics"      = "true"
    "--SOURCE_BUCKET"       = var.source_bucket_name
    "--DEST_BUCKET"         = aws_s3_bucket.destination_bucket.bucket
  }

  glue_version      = "3.0"
  number_of_workers = 2
  worker_type       = "G.1X"
  max_retries       = 1

  tags = {
    Environment = var.environment
  }
}

############################################################
# 6. AWS Glue Crawler (us-east-2)
############################################################

resource "aws_glue_crawler" "iceberg_crawler" {
  name          = "iceberg-crawler-${var.environment}"
  role          = aws_iam_role.glue_role.arn
  database_name = aws_glue_catalog_database.iceberg_db.name

  s3_target {
    path = "s3://${aws_s3_bucket.destination_bucket.bucket}/iceberg-data/"
  }

  configuration = jsonencode({
    Version = 1.0,
    CrawlerOutput = {
      Partitions = {
        AddOrUpdateBehavior = "InheritFromTable"
      }
    }
  })

  schedule = "cron(0 3 * * ? *)"

  tags = {
    Environment = var.environment
  }
}

Error (Account and Request ID replaced with random values)

An error occurred while calling o111.createOrReplace. User: arn:aws:sts::1111:assumed-role/glue-cross-region-role-dev/GlueJobRunnerSession is not authorized to perform: glue:GetTable on resource: arn:aws:glue:us-west-1:1111:catalog because no identity-based policy allows the glue:GetTable action (Service: Glue, Status Code: 400, Request ID: 12121212)

We really don't require to operate on the DB within us-west-1 region, but we already have created the DB via terraform script in us-east-2. How can we overcome this issue and make sure that glue job should only run process in us-west-1 and transfer data to us-east-2??

Any help would be appreciated to solve this issue. We had a hard requirement to use Iceberg as the format