Hello,
I'm using Hudi connector in Glue, first, I bulk inserted the initial dataset to Hudi table, I'm adding a daily incremental records and I can query them using Athena, what I'm trying to do is to get the newly added, updated or deleted records in a separate parquet file.
I've tried different approaches and configurations using both copy on write and merge on read tables but still can get the updates in a separate file.
I used these configurations in different combinations:
'className' : 'org.apache.hudi',
'hoodie.datasource.hive_sync.use_jdbc': 'false',
'hoodie.datasource.write.precombine.field': 'ts',
'hoodie.datasource.write.recordkey.field': 'uuid',
'hoodie.payload.event.time.field': 'ts',
'hoodie.table.name': 'table_name',
'hoodie.datasource.hive_sync.database': 'hudi_db',
'hoodie.datasource.hive_sync.table': 'table_name',
'hoodie.datasource.hive_sync.enable': 'false',
# 'hoodie.datasource.write.partitionpath.field': 'date:SIMPLE',
'hoodie.datasource.write.hive_style_partitioning': 'true',
'hoodie.meta.sync.client.tool.class': 'org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool',
'hoodie.datasource.write.table.type': 'COPY_ON_WRITE',
'path': 's3://path/to/output/',
# 'hoodie.datasource.write.operation': 'bulk_insert',
'hoodie.datasource.write.operation': 'upsert',
# 'hoodie.datasource.hive_sync.partition_extractor_class': 'org.apache.hudi.hive.NonPartitionedExtractor',
# 'hoodie.datasource.hive_sync.partition_extractor_class': 'org.apache.hudi.hive.MultiPartKeysValueExtractor',
'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.NonpartitionedKeyGenerator',
# 'hoodie.compaction.payload.class': 'org.apache.hudi.common.model.OverwriteWithLatestAvroPayload',
# 'hoodie.cleaner.policy': 'KEEP_LATEST_COMMITS',
'hoodie.cleaner.delete.bootstrap.base.file': 'true',
"hoodie.index.type": "GLOBAL_BLOOM",
'hoodie.file.index.enable': 'true',
'hoodie.bloom.index.update.partition.path': 'true',
'hoodie.bulkinsert.shuffle.parallelism': 1,
# 'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.CustomKeyGenerator'
Thank you.