AWS Crawler to directly read Delta lake files from S3

0

Are there any ways to read delta lake files from s3 and create Data catalog on top of this to run Glue ETL job? When I crawl in delta folders it creates separate schema for log, manifest & parquets rather then each tables with all the delta log, manifest files and parquet files , https://docs.aws.amazon.com/glue/latest/dg/crawler-data-stores.html says Crawler now have native client then why it is not recognizing the path , Am I missing something.

preguntada hace 2 años452 visualizaciones
1 Respuesta
0

Hi,

Please try installing the Delta Lake Connector for AWS Glue which can be found here.

https://aws.amazon.com/marketplace/pp/prodview-seypofzqhdueq

The Delta Lake Connector will allow you to connect to Delta Lake tables from your Glue jobs and it is offered at no additional cost.

Hope this helps.

profile pictureAWS
respondido hace 2 años
profile picture
INGENIERO DE SOPORTE
revisado hace 2 años

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas