AWS Data Catalog table index not working

Question

Hello,

For an AWS Data Catalog table, I ran Glue (structure: Amazon S3 -> Change Schema -> AWS Glue Data Catalog ) and populate table with only string records.  All the actions were done from the Management Console: table, partition, index creation, ETL visual creation, ETL run.
  
After checking that the records exists in the table, I created a partition and an index. After this, all the table rows were deleted, and COUNT from Athena returned 0.  
  
If I tried to create the partition and index before the upload of the data with Glue, no data was inserted.  
  
Which are the rules for creating partitions and indexes in a Data Catalog table ?  
  
Thank you,  
Mihai

Accepted Answer

The index is just speeding up the partition lookup, it won't affect the data. 
The partition columns do affect how the data is structured on s3 and how is inserted.  
Normally you define the partition columns/index and  the tool writing into the table will register the partitions after inserting the data, if the partitions are not registered with specific values and paths then the data is not available for querying.

AWS Data Catalog table index not working

관련 콘텐츠