AWS Data Catalog table index not working

Hello,

For an AWS Data Catalog table, I ran Glue (structure: Amazon S3 -> Change Schema -> AWS Glue Data Catalog ) and populate table with only string records. All the actions were done from the Management Console: table, partition, index creation, ETL visual creation, ETL run.

After checking that the records exists in the table, I created a partition and an index. After this, all the table rows were deleted, and COUNT from Athena returned 0.

If I tried to create the partition and index before the upload of the data with Glue, no data was inserted.

Which are the rules for creating partitions and indexes in a Data Catalog table ?

Thank you,
Mihai

トピック

分析データベース

タグ

AWS Glue データの抽出、変換、ロード

言語

English

rePost-User-4717319

質問済み 4ヶ月前156ビュー

1回答

新しい順
投票が多い順
コメントが多い順

これらの回答は役に立ちましたか？コミュニティがあなたの知識を活用できるように、正解に賛成票を投じてください。

承認された回答

The index is just speeding up the partition lookup, it won't affect the data. The partition columns do affect how the data is structured on s3 and how is inserted.
Normally you define the partition columns/index and the tool writing into the table will register the partitions after inserting the data, if the partitions are not registered with specific values and paths then the data is not available for querying.

エキスパート

Gonzalo Herreros

回答済み 4ヶ月前

エキスパート

Didier_Durand

レビュー済み 4ヶ月前

rePost-User-4717319
3ヶ月前
Thanks a lot,

After deleting the partition and index, I can see the data in the table again.

If I used as a tool to write into the table, Glue ETL, how should I register the partition and values/paths in the above configuration (should I use a specific component or should I be able to do this from the AWS Glue Data Catalog component) ?

When you said speeding the lookup, does this means also to decrease the scanned volume of data (instead of scanning data, to scan the index) ?

Thank you, Mihai

AWS Data Catalog table index not working

関連するコンテンツ