AWS Crawler to directly read Delta lake files from S3

0

Are there any ways to read delta lake files from s3 and create Data catalog on top of this to run Glue ETL job? When I crawl in delta folders it creates separate schema for log, manifest & parquets rather then each tables with all the delta log, manifest files and parquet files , https://docs.aws.amazon.com/glue/latest/dg/crawler-data-stores.html says Crawler now have native client then why it is not recognizing the path , Am I missing something.

posta 2 anni fa452 visualizzazioni
1 Risposta
0

Hi,

Please try installing the Delta Lake Connector for AWS Glue which can be found here.

https://aws.amazon.com/marketplace/pp/prodview-seypofzqhdueq

The Delta Lake Connector will allow you to connect to Delta Lake tables from your Glue jobs and it is offered at no additional cost.

Hope this helps.

profile pictureAWS
con risposta 2 anni fa
profile picture
TECNICO DI SUPPORTO
verificato 2 anni fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande