Delta table column mapping support in Glue/Athena

0

I'm confused by AWS documentation regarding compatibility with delta tables. We need to delete a column that is the "column mapping" feature supported in delta-lake 1.2.0 and we do it through spark sql and it's mandatory to to specify parameters for our table in S3 in order to do this:

'delta.columnMapping.mode' = 'name',
'delta.minReaderVersion' = '2',
'delta.minWriterVersion' = '5'

https://docs.delta.io/latest/delta-column-mapping.html

After doing this we want to update our table in Glue but the crawler fails because it uses Glue version 3 for crawling and isn't compatible with delta lake 1.2.0. https://repost.aws/questions/QUyDYz31OnREGxy7gz2qIeuw/error-internal-service-exception-of-glue-crawler

We tried to create the table through Athena that states:

Column mapping and timestampNtz – Delta column mapping, which allows Delta table columns and the underlying Parquet file columns to use different names, and timestamp without timezone (timestampNtz) are supported.

Delta Lake reader version – Delta Lake reader protocol up to version 3 is supported.

(nothing about the writer though)

https://docs.aws.amazon.com/athena/latest/ug/delta-lake-tables.html

But it fails with:

Delta protocol version is too new for Athena DDL engine

So what are the options to update our table in Glue to be usable with Athena?

Sergii
已提问 2 个月前256 查看次数
1 回答
0

It seems like you’re facing an issue because the Delta protocol version is too new for the Athena DDL engine. One possible solution could be to rerun the crawler to update the Delta table in the AWS Glue Data Catalog.

Look at this ref: https://delta.io/blog/delta-lake-aws-glue/

profile picture
专家
已回答 2 个月前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则