- 新しい順
- 投票が多い順
- コメントが多い順
Hi,
I understand that you are trying to access tables from two different glue catalogue accounts using a glue job. We can setup the access policies in source and target accounts and then use two different dynamic frames to access these tables. We don’t need to use the “--conf spark.hadoop.hive.metastore.glue.catalogid” option for your use case. The step by step process to setup for cross account is provided here -
https://repost.aws/knowledge-center/glue-tables-cross-accounts
We can first setup the cross account permissions from Account A [Catalogue account] to Account B and Account C separately. Then in the glue job we can create two dynamic frames [df1 and df2] to access tables from the accounts B and C.
For Example :
df1 = glueContext.create_dynamic_frame.from_catalog(database="doc_example_DB", table_name="doc_example_table", catalog_id=“Account B”, region="us-east-1")
df2 = glueContext.create_dynamic_frame.from_catalog(database="doc_example_DB", table_name="doc_example_table", catalog_id="Account C", region="us-east-1")
Thank you.
Thank you very much. This helps a lot
df1 = glueContext.create_dynamic_frame.from_catalog(database="doc_example_DB", table_name="doc_example_table", catalog_id=“Account B”, region="us-east-1")
df2 = glueContext.create_dynamic_frame.from_catalog(database="doc_example_DB", table_name="doc_example_table", catalog_id="Account C", region="us-east-1")
Above format doesnt work if the table is of iceberg format in catalog.
IllegalArgumentException: getDynamicFrame is not supported for iceberg table. Please use getDataFrame instead.