1 Antwort
- Neueste
- Die meisten Stimmen
- Die meisten Kommentare
1
Sounds you are creating a table that is not really Delta but just reading the parquet files.
For native format you just specify the location and type: https://docs.aws.amazon.com/athena/latest/ug/delta-lake-tables.html
and I don't think you can use the REPAIR, you would have to use a crawler or synchronize using the tool provided by Delta.
Relevanter Inhalt
- AWS OFFICIALAktualisiert vor 2 Jahren
- AWS OFFICIALAktualisiert vor einem Jahr
- Wie behebe ich den Fehler „java.lang.OutOfMemoryError: Java heap space“ in einem AWS Glue-Spark-Job?AWS OFFICIALAktualisiert vor 2 Jahren
- AWS OFFICIALAktualisiert vor 2 Jahren
Thanks Gonzalo! Makes sense. The DDL you linked works from within Athena, but not from within Spark.
Do you know how to create a native delta table for Athena engine v3 in PySpark?
I think it's doable, in Spark write the data with the delta library and then create the table in the catalog using CREATE EXTERNAL TABLE specifying the columns, location and 'table_type'='DELTA'