Glue table partition format question

0

Hi, We send our database old data into S3 in this format:

{table}/year/month/day/asset_id.parquet so we have a lot of files like this: table/2024/01/28/z.parquet table/2024/01/28/y.parquet table/2024/01/29/z.parquet table/2024/01/29/y.parquet etc...

When I edit my glue table and click on Partition it says : "No available partitions." Am I missing something? Is it because we need to specify in the file name the actual meaning of the number like here for example ? {table}/year=2024/month=10/day=28/z.parquet {table}/year=2024/month=10/day=28/y.parquet

Is Glue able to automatically partition with my file naming convention "accelerometer/2024/01/28" ?

What do you guys recommend doing for me ? Thank you!

LouisAW
質問済み 3ヶ月前362ビュー
1回答
0

The partitions have to be explicitly added to the catalog (expect from Athena projections), is not enough to have the data in the right path
Because you are not following the naming convention, you cannot "repair" the table and the crawler cannot give it meaning partition names (will be named 0, 1, 2...) but you could rename the partition columns after the crawler has loaded them.

profile pictureAWS
エキスパート
回答済み 3ヶ月前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ