Issues creating Glue Dynamic Frame from custom catalog using Athena Timestream connector

0

Hi all,

I have followed the instructions https://docs.aws.amazon.com/athena/latest/ug/connect-data-source-serverless-app-repo.html to deploy Timestream as an additional data source to Athena and can succeassfully query timestream data via Athena console, by using catalog "TimestreamCatalog" I added.

Now I need to use the same catalog "TimestreamCatalog" when building a Glue job.

I run:

DataCatalogtable_node1 = glueContext.create_dynamic_frame.from_catalog(
    catalog_id = "TimestreamCatalog",
    database="mydb",
    table_name="mytable",
    transformation_ctx="DataCatalogtable_node1",
)

and run into this error, even when the role in question has Administrator policy i.e. action:* resource* attached (for the sake of experiment):

An error occurred while calling o86.getCatalogSource. User: arn:aws:sts::*******:assumed-role/AWSGlueServiceRole-andrei/GlueJobRunnerSession is not authorized to perform: glue:GetTable on resource: arn:aws:glue:eu-central-1:TimestreamCatalog:catalog (Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException; Request ID: 36d7e411-8ca9-4993-9066-b6ca1d7ea4a3; Proxy: null)

When calling aws athena list-data-catalogs , I get:


{
    "DataCatalogsSummary": [
        {
            "CatalogName": "AwsDataCatalog",
            "Type": "GLUE"
        },
        {
            "CatalogName": "TimestreamCatalog",
            "Type": "LAMBDA"
        }
    ]
}

I am not sure if using data source name as catalog_id is correct here, so any hint on what catalog_id is supposed to be for customer data source is appreciated, or any hint on how to resolve the issue above.

Thanks, Andrei

質問済み 1年前316ビュー
1回答
0
  1. When you create the connector and query the data in Athena , Athena just queries the underlying datasource to give you the data . It does not extract the data and store in Athena itself , unless you do CTAS . When querying from Glue, try to read the actual datasource instead of Athena .
  2. You can also use the glue visual developer to see if it works and switch to the script mode to copy the code and use in your original job.
  3. Alternatively , in your glue code, use Boto3 to query the Athena table , store the results in a S3 file and read it to a DF
回答済み 1年前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ