Issues creating Glue Dynamic Frame from custom catalog using Athena Timestream connector

0

Hi all,

I have followed the instructions https://docs.aws.amazon.com/athena/latest/ug/connect-data-source-serverless-app-repo.html to deploy Timestream as an additional data source to Athena and can succeassfully query timestream data via Athena console, by using catalog "TimestreamCatalog" I added.

Now I need to use the same catalog "TimestreamCatalog" when building a Glue job.

I run:

DataCatalogtable_node1 = glueContext.create_dynamic_frame.from_catalog(
    catalog_id = "TimestreamCatalog",
    database="mydb",
    table_name="mytable",
    transformation_ctx="DataCatalogtable_node1",
)

and run into this error, even when the role in question has Administrator policy i.e. action:* resource* attached (for the sake of experiment):

An error occurred while calling o86.getCatalogSource. User: arn:aws:sts::*******:assumed-role/AWSGlueServiceRole-andrei/GlueJobRunnerSession is not authorized to perform: glue:GetTable on resource: arn:aws:glue:eu-central-1:TimestreamCatalog:catalog (Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException; Request ID: 36d7e411-8ca9-4993-9066-b6ca1d7ea4a3; Proxy: null)

When calling aws athena list-data-catalogs , I get:


{
    "DataCatalogsSummary": [
        {
            "CatalogName": "AwsDataCatalog",
            "Type": "GLUE"
        },
        {
            "CatalogName": "TimestreamCatalog",
            "Type": "LAMBDA"
        }
    ]
}

I am not sure if using data source name as catalog_id is correct here, so any hint on what catalog_id is supposed to be for customer data source is appreciated, or any hint on how to resolve the issue above.

Thanks, Andrei

질문됨 일 년 전316회 조회
1개 답변
0
  1. When you create the connector and query the data in Athena , Athena just queries the underlying datasource to give you the data . It does not extract the data and store in Athena itself , unless you do CTAS . When querying from Glue, try to read the actual datasource instead of Athena .
  2. You can also use the glue visual developer to see if it works and switch to the script mode to copy the code and use in your original job.
  3. Alternatively , in your glue code, use Boto3 to query the Athena table , store the results in a S3 file and read it to a DF
답변함 일 년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인