By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Hudi and S3 object versions

0

We are currently using HUDI for upserts via AWS Glue. We also have S3 bucket versioning applied on the bucket where HUDI is running on. My question is related to previous versions of objects, if we delete the previous version of an object will that impact the HUDI operation or is it the case that HUDI only works on the latest version of the object?

Topics
Analytics
Tags
AWS Glue
Language
English
GordonFV
asked 2 years ago439 views
1 Answer
0

Per my understanding, HUDI works on the latest version of the S3 object. Hence deleting an older version of object shouldn't cause issues with HUDI upserts. You may want to look into S3 delete markers and deleting object version ID in that case - https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeletingObjectVersions.html

AWS
Meghana_S
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content