Delta table column mapping support in Glue/Athena

0

I'm confused by AWS documentation regarding compatibility with delta tables. We need to delete a column that is the "column mapping" feature supported in delta-lake 1.2.0 and we do it through spark sql and it's mandatory to to specify parameters for our table in S3 in order to do this:

'delta.columnMapping.mode' = 'name',
'delta.minReaderVersion' = '2',
'delta.minWriterVersion' = '5'

https://docs.delta.io/latest/delta-column-mapping.html

After doing this we want to update our table in Glue but the crawler fails because it uses Glue version 3 for crawling and isn't compatible with delta lake 1.2.0. https://repost.aws/questions/QUyDYz31OnREGxy7gz2qIeuw/error-internal-service-exception-of-glue-crawler

We tried to create the table through Athena that states:

Column mapping and timestampNtz – Delta column mapping, which allows Delta table columns and the underlying Parquet file columns to use different names, and timestamp without timezone (timestampNtz) are supported.

Delta Lake reader version – Delta Lake reader protocol up to version 3 is supported.

(nothing about the writer though)

https://docs.aws.amazon.com/athena/latest/ug/delta-lake-tables.html

But it fails with:

Delta protocol version is too new for Athena DDL engine

So what are the options to update our table in Glue to be usable with Athena?

Sergii
preguntada hace 2 meses255 visualizaciones
1 Respuesta
0

It seems like you’re facing an issue because the Delta protocol version is too new for the Athena DDL engine. One possible solution could be to rerun the crawler to update the Delta table in the AWS Glue Data Catalog.

Look at this ref: https://delta.io/blog/delta-lake-aws-glue/

profile picture
EXPERTO
respondido hace 2 meses

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas