Questo contenuto non è disponibile nella lingua selezionata

Lavoriamo costantemente per rendere disponibili i contenuti nella lingua selezionata. Ti ringraziamo per la comprensione.

How can I set up a cross-account AWS Glue ETL job for tables using catalog resource policies?

4 minuti di lettura

I want to access tables in account A from account B using ETL jobs in the AWS Glue Data Catalog, without using AWS Lake Formation.

Short description

You can access cross-account resources in the AWS Glue Data Catalog with or without AWS Lake Formation. The following sections outline the setup to access a cross-account catalog using only AWS Glue.

If you use Lake Formation, see Cross-account data sharing in Lake Formation for more information on how to set up a cross-account resource share.

Note: The following steps describe how to access cross-account resources within a single AWS Region. They don't address access to resources in a different Region.

Resolution

Set up access policies in source and target accounts

Use the following steps to grant resource-level permissions to account B from account A's AWS Glue Data Catalog.

Note: Account A has the AWS Glue Data Catalog resources and account B is the extract, transform, and load (ETL) account. Account A has the resource-based policy modifications and account B holds the AWS Identity and Access Management (IAM) policy modifications.

Attach a catalog-resource policy in account A

Open the AWS Glue console.
In the navigation pane, choose Catalog settings.

Under Permissions, enter the following resource policy. This resource policy lets account B access the databases and tables in account A.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:root"
      },
      "Action": "glue:*",
      "Resource": [
        "arn:aws:glue:us-east-1:555566667777:catalog",
        "arn:aws:glue:us-east-1:555566667777:database/doc_example_DB",
        "arn:aws:glue:us-east-1:555566667777:table/doc_example_DB/*"
      ]
    }
  ]
}

Note: The preceding example policy includes a wildcard (glue:*) that grants all AWS Glue permissions to account B on the specified resources. To follow the principle of least privilege, replace the wildcard with only the specific permissions that your use case requires, such as glue:GetDatabase, glue:GetTable, and glue:GetPartition. Replace 111122223333 with the account ID for account B, 555566667777 with the account ID for account A, us-east-1 with your Region, and doc_example_DB with the name of your database.

(Optional) You can limit access to a specific role in account B by including the Amazon Resource Name (ARN) of the role in the policy. For example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/service-role/AWSGlueServiceRole_Glue_Test"
      },
      "Action": "glue:*",
      "Resource": [
        "arn:aws:glue:us-east-1:555566667777:catalog",
        "arn:aws:glue:us-east-1:555566667777:database/doc_example_DB",
        "arn:aws:glue:us-east-1:555566667777:table/doc_example_DB/*"
      ]
    }
  ]
}

Note: The example policy includes the glue:* wildcard. To follow the principle of least privilege, replace the wildcard with only the specific permissions that your use case requires. Replace 111122223333 with the account ID for account B, 555566667777 with the account ID for account A, us-east-1 with your Region, doc_example_DB with the name of your database, and AWSGlueServiceRole_Glue_Test with the name of the role that runs the ETL job.

Attach an IAM policy in account B

The IAM role in account B that runs the ETL job needs access to the databases and tables in account A.

Note: If you use Amazon Athena with the Data Catalog, then include the default database in the policy. This makes sure that the GetDatabase and CreateDatabase actions succeed. For more information, see Default database and catalog per AWS Region.

Open the IAM console.
In the navigation pane, choose Roles.
Choose the role name that you use for the ETL job.

Attach an IAM policy to the AWS Glue ETL job's IAM role in account B. This gives you access to the database and tables in account A:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetDatabase",
        "glue:GetConnection",
        "glue:GetTable",
        "glue:GetPartition"
      ],
      "Resource": [
        "arn:aws:glue:us-east-1:555566667777:catalog",
        "arn:aws:glue:us-east-1:555566667777:database/default",
        "arn:aws:glue:us-east-1:555566667777:database/doc_example_DB",
        "arn:aws:glue:us-east-1:555566667777:table/doc_example_DB/*"
      ]
    }
  ]
}

Note: Replace 555566667777 with the account ID for account A, us-east-1 with your Region, and doc_example_DB with the name of your database.

Verify that the policy you created is attached to the IAM role in account B.

Test if account B has access to the Data Catalog in account A. Create an ETL job with the following scripts:

Dynamic frame script:

df = glueContext.create_dynamic_frame.from_catalog(database="doc_example_DB", table_name="doc_example_table", catalog_id="555566667777", region="us-east-1")

Data frame script:

"""Create Spark Session with cross-account AWS Glue Data Catalog"""
from pyspark.sql import SparkSession

spark_session = SparkSession.builder.appName("Spark Glue Example") \
.config("hive.metastore.client.factory.class", \
"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory") \
.config("hive.metastore.glue.catalogid", "555566667777") \
.enableHiveSupport() \
.getOrCreate()

table_df = spark_session.sql("SELECT * FROM doc_example_DB.doc_example_table limit 10")

table_df.show()

Note: Replace 555566667777 with the account ID for account A, doc_example_DB with the name of your database, doc_example_table with the name of your table, and us-east-1 with your Region.