AWS Glue Notebook issue when running SQL script
I am following the steps outlined in the link below:
(1) No issue with Query Delta Lake tables using Amazon Athena, able to query out the data.
(2) Issue AWS Glue Notebook:
(a) created IAM role named "AWSGlueServiceRoleDefault" which includes
(i) AmazonS3FullAccess, which is AWS managed
(ii) AWSGlueServiceRole, which is AWS managed
(iii) PassRolePolicy, which is Customer Inline
PassRolePolicy as below:
Following the instructions from the link, which I used the IAM role ("AWSGlueServiceRoleDefault") I created above, the first part of the Python code is able to run as shown below:
But the SQL portion throws me a list of Py4JJavaError:
Even the below Python code which tries to retrieve information about the table generates the same error as below:
The above error is repeatable, which can be viewed here: https://justpaste.it/1zxzz
Would appreciate if you could help me to see which portion is creating the error, and how to remedy it. Thanks.
- 最新
- 投票最多
- 评论最多
The libraries for the Delta format are not enabled by default.
The job needs a parameter: --datalake-formats=delta
From a notebook you need to pass it as config before you run any other cell (once the session is started, it won't take effect)
%%configure
{
"--datalake-formats": "delta"
}
Splendid.... your short and concise answer saves me lots of hours of frustration to find out exactly what is the issue. Thank you very much.
Take note above also needs to add IAM:PassRole policy, which is not instructed in the first link, else it would have issue relating to IAM:PassRole.