Delta table column mapping support in Glue/Athena

0

I'm confused by AWS documentation regarding compatibility with delta tables. We need to delete a column that is the "column mapping" feature supported in delta-lake 1.2.0 and we do it through spark sql and it's mandatory to to specify parameters for our table in S3 in order to do this:

'delta.columnMapping.mode' = 'name',
'delta.minReaderVersion' = '2',
'delta.minWriterVersion' = '5'

https://docs.delta.io/latest/delta-column-mapping.html

After doing this we want to update our table in Glue but the crawler fails because it uses Glue version 3 for crawling and isn't compatible with delta lake 1.2.0. https://repost.aws/questions/QUyDYz31OnREGxy7gz2qIeuw/error-internal-service-exception-of-glue-crawler

We tried to create the table through Athena that states:

Column mapping and timestampNtz – Delta column mapping, which allows Delta table columns and the underlying Parquet file columns to use different names, and timestamp without timezone (timestampNtz) are supported.

Delta Lake reader version – Delta Lake reader protocol up to version 3 is supported.

(nothing about the writer though)

https://docs.aws.amazon.com/athena/latest/ug/delta-lake-tables.html

But it fails with:

Delta protocol version is too new for Athena DDL engine

So what are the options to update our table in Glue to be usable with Athena?

Sergii
已提問 2 個月前檢視次數 258 次
1 個回答
0

It seems like you’re facing an issue because the Delta protocol version is too new for the Athena DDL engine. One possible solution could be to rerun the crawler to update the Delta table in the AWS Glue Data Catalog.

Look at this ref: https://delta.io/blog/delta-lake-aws-glue/

profile picture
專家
已回答 2 個月前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南