使用 AWS re:Post 即表示您同意 AWS re:Post 使用條款
AWS re:Postre:Post

AWS Data Catalog table index not working

0

Hello,

For an AWS Data Catalog table, I ran Glue (structure: Amazon S3 -> Change Schema -> AWS Glue Data Catalog ) and populate table with only string records. All the actions were done from the Management Console: table, partition, index creation, ETL visual creation, ETL run.

After checking that the records exists in the table, I created a partition and an index. After this, all the table rows were deleted, and COUNT from Athena returned 0.

If I tried to create the partition and index before the upload of the data with Glue, no data was inserted.

Which are the rules for creating partitions and indexes in a Data Catalog table ?

Thank you,
Mihai

主題
分析資料庫
標籤
AWS Glue提取轉換與載入資料
語言
English
rePost-User-4717319
已提問 4 個月前檢視次數 156 次
1 個回答
0
已接受的答案

The index is just speeding up the partition lookup, it won't affect the data. The partition columns do affect how the data is structured on s3 and how is inserted.
Normally you define the partition columns/index and the tool writing into the table will register the partitions after inserting the data, if the partitions are not registered with specific values and paths then the data is not available for querying.

profile pictureAWS
專家
Gonzalo Herreros
已回答 4 個月前
profile pictureAWS
專家
Didier_Durand
已審閱 4 個月前
  • rePost-User-4717319
    3 個月前

    Thanks a lot,

    After deleting the partition and index, I can see the data in the table again.

    If I used as a tool to write into the table, Glue ETL, how should I register the partition and values/paths in the above configuration (should I use a specific component or should I be able to do this from the AWS Glue Data Catalog component) ?

    When you said speeding the lookup, does this means also to decrease the scanned volume of data (instead of scanning data, to scan the index) ?

    Thank you, Mihai

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋，同時有助於提問者的專業成長。

回答問題指南

相關內容