bucket size calculation issue

Question

I am facing issue when i am calculating size of s3 bucket . when i check from the bucket metrics , it shows 54TB , but when i calculate the total size of bucket through api/console it shows only 1 TB. we haven't enabled any versioning. 
the cost is also calculate on basis of 54 TB . please help me with that. 
this bucket is used for datalake operation and there are lot of read/write/deletion operation happen over the time.

Answer

First of all, I suggest enabled S3 Storage Lens. The default parameters are free and take 24 hours for the data to show up. This can be used to determine what is causing the size in the bucket.

In addition to versioning, the other factor that S3 Storage Lens can show you is if you have any lingering Multi-Part Uploads (MPU). These are remnants of objects that may be left only partially uploaded and take up storage space. I recommend creating a Lifecycle Policy on your buckets that will remove MPUs after some reasonable amount of time (1-2 weeks).

* https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage_lens.html
* https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
* https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-abort-incomplete-mpu-lifecycle-config.html

Answer

You should also open a support ticket with AWS and ask for a refund of the costs for the multipart uploads that you paid for.  Be prepared to tell them which buckets and how far back for them to go.

bucket size calculation issue

Relevant content