Issues creating Glue Dynamic Frame from custom catalog using Athena Timestream connector

0

Hi all,

I have followed the instructions https://docs.aws.amazon.com/athena/latest/ug/connect-data-source-serverless-app-repo.html to deploy Timestream as an additional data source to Athena and can succeassfully query timestream data via Athena console, by using catalog "TimestreamCatalog" I added.

Now I need to use the same catalog "TimestreamCatalog" when building a Glue job.

I run:

DataCatalogtable_node1 = glueContext.create_dynamic_frame.from_catalog(
    catalog_id = "TimestreamCatalog",
    database="mydb",
    table_name="mytable",
    transformation_ctx="DataCatalogtable_node1",
)

and run into this error, even when the role in question has Administrator policy i.e. action:* resource* attached (for the sake of experiment):

An error occurred while calling o86.getCatalogSource. User: arn:aws:sts::*******:assumed-role/AWSGlueServiceRole-andrei/GlueJobRunnerSession is not authorized to perform: glue:GetTable on resource: arn:aws:glue:eu-central-1:TimestreamCatalog:catalog (Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException; Request ID: 36d7e411-8ca9-4993-9066-b6ca1d7ea4a3; Proxy: null)

When calling aws athena list-data-catalogs , I get:


{
    "DataCatalogsSummary": [
        {
            "CatalogName": "AwsDataCatalog",
            "Type": "GLUE"
        },
        {
            "CatalogName": "TimestreamCatalog",
            "Type": "LAMBDA"
        }
    ]
}

I am not sure if using data source name as catalog_id is correct here, so any hint on what catalog_id is supposed to be for customer data source is appreciated, or any hint on how to resolve the issue above.

Thanks, Andrei

已提问 1 年前316 查看次数
1 回答
0
  1. When you create the connector and query the data in Athena , Athena just queries the underlying datasource to give you the data . It does not extract the data and store in Athena itself , unless you do CTAS . When querying from Glue, try to read the actual datasource instead of Athena .
  2. You can also use the glue visual developer to see if it works and switch to the script mode to copy the code and use in your original job.
  3. Alternatively , in your glue code, use Boto3 to query the Athena table , store the results in a S3 file and read it to a DF
已回答 1 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则