Empty bucket not really empty?

Question

Hi!

We are using this particular bucket as a temporary storage - one service stores big (1-500GB) dump files there and another scheduled daily job checks if this bucket contains any files and retrieves them (files are removed from the bucket afterwards). It's a really low frequency traffic and there shouldn't be more than a couple of these files per day - there are days with no traffic at all. Versioning is off, of course.

And this is the problem: this bucket is empty (for days now), but if I look at bucket metrics, CloudWatch claims there is around 1TB of data there in more than 65K of files. I understand that sometimes operations are just scheduled and do not happen immediately, but a week seems like enough time to delete a couple of files.

What have I tried so far?
1. I read somewhere that these files could be remnants from failed uploads, which are not shown because they are not complete files. And that they could be cleared out if I turned the versioning on with a lifecycle rule that deletes everything. Tried, it didn't change anything.
2. I deleted this bucket completely - it shouldn't work if bucket is not empty in the first place (as far as I understand). I created a new bucket with the same name after a day and lo and behold - total bucket size is 1018GB, according to metrics.

If there are files still in this bucket, how do I see/remove them from this limbo? If that's not the case, how do I stop CloudWatch from giving me false information? Why pay for "nothing"?

Cheers

Answer

This could be related to Amazon S3's multipart upload feature. If the complete multipart upload request isn’t sent successfully, Amazon S3 will not assemble the parts and will not create any object. The parts remain in your Amazon S3 account until the multipart upload completes or is aborted, and you pay for the parts that are stored in Amazon S3. Also note that you aren’t able to view the parts of your incomplete multipart upload in the AWS Management Console.

While it is possible to manually list and abort incomplete multipart uploads in your S3 buckets, this can quickly become a chore.

Consider this solution from AWS. There are numerous others if you search if your problem is actually related to aborted multipart uploads.

While it is possible to manually [list](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/list-multipart-uploads.html) and [abort](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/abort-multipart-upload.html) incomplete multipart uploads in your S3 buckets, this can quickly become a chore.

Consider this solution from [AWS](https://aws.amazon.com/blogs/aws-cloud-financial-management/discovering-and-deleting-incomplete-multipart-uploads-to-lower-amazon-s3-costs/). There are numerous others if you search if your problem is actually related to aborted multipart uploads.

Answer

Thanks RoB,

Yeah, as I mentioned, I have tried this AWS recipe with lifecycle rule (although I ticked all boxes, to delete everything there, not just deleted markers and incomplete multipart uploads), but it didn't solve the issue. I'm gonna try with manual procedure and report how it went.

Answer

Having the same issue.  22.2 GB was being reported for one of my buckets I used for a multiport upload that failed.  I deleted the bucket (as stated above should not happen if there is something in the bucket).
I have no information anymore on that multipart upload, it was a while ago, a few months.
Would really like to know what the answer is.

Empty bucket not really empty?

Relevant content