AWS S3 batch operation with different total objects between buckets

0

Hello,

I have executed a batch operation to replicate the objects from one bucket to another, but from the Metrics tab, the number of objects on the destination bucket is different from the amount of objects migrated.

Source bucket total objects: 35869464 Batch job total objects migrated: 35827228 Destination bucket total objects: 657373

The buckets have versioning enabled.

What can explain these differences?

asked a year ago648 views
1 Answer
0

You may have multipart uploads that failed and are still in your buckets. Check out these two articles on that situation. You will have to create a lifecyle rule for each bucket to get rid of these. I would use 1 day for the time period or else you'll have to wait a long time for it to take effect. Storage lens will tell you if you have these failed uploads or not. You won't see them in your buckets.

https://aws.amazon.com/blogs/aws-cloud-financial-management/discovering-and-deleting-incomplete-multipart-uploads-to-lower-amazon-s3-costs/

https://aws.amazon.com/blogs/aws/s3-lifecycle-management-update-support-for-multipart-uploads-and-delete-markers/

You might want to open a support ticket with AWS to get a refund for these failed uploads. I did and they did refund me for them. Be prepared to tell them what buckets and how far back to go for the refund.

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions