Delta table column mapping support in Glue/Athena

0

I'm confused by AWS documentation regarding compatibility with delta tables. We need to delete a column that is the "column mapping" feature supported in delta-lake 1.2.0 and we do it through spark sql and it's mandatory to to specify parameters for our table in S3 in order to do this:

'delta.columnMapping.mode' = 'name',
'delta.minReaderVersion' = '2',
'delta.minWriterVersion' = '5'

https://docs.delta.io/latest/delta-column-mapping.html

After doing this we want to update our table in Glue but the crawler fails because it uses Glue version 3 for crawling and isn't compatible with delta lake 1.2.0. https://repost.aws/questions/QUyDYz31OnREGxy7gz2qIeuw/error-internal-service-exception-of-glue-crawler

We tried to create the table through Athena that states:

Column mapping and timestampNtz – Delta column mapping, which allows Delta table columns and the underlying Parquet file columns to use different names, and timestamp without timezone (timestampNtz) are supported.

Delta Lake reader version – Delta Lake reader protocol up to version 3 is supported.

(nothing about the writer though)

https://docs.aws.amazon.com/athena/latest/ug/delta-lake-tables.html

But it fails with:

Delta protocol version is too new for Athena DDL engine

So what are the options to update our table in Glue to be usable with Athena?

Sergii
demandé il y a 2 mois258 vues
1 réponse
0

It seems like you’re facing an issue because the Delta protocol version is too new for the Athena DDL engine. One possible solution could be to rerun the crawler to update the Delta table in the AWS Glue Data Catalog.

Look at this ref: https://delta.io/blog/delta-lake-aws-glue/

profile picture
EXPERT
répondu il y a 2 mois

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions