How to delete a bucket with millions of objects?

It's normal for the object count to increase when objects are deleted. That's because in a versioning-enabled bucket, a regular object deletion won't actually delete anything but rather create a new, zero-byte-sized "delete marker", which is sort of like a tombstone that covers the object that used to exist but is no longer with us. The delete marker counts as an additional object but doesn't significantly increase the total bucket size, because it's an empty object.

In general, to delete the contents of any bucket permanently, regardless of whether versioning is enabled, disabled, or suspended, you'd want to create three rules (or combine them if you'd like):

1. "delete-current-versions": delete current versions of objects 1 day after creation
2. "delete-noncurrent-versions": delete non-current versions of objects 1 day after becoming noncurrent, retaining 0 versions
3. "daily-housekeeping": delete expired delete markers, and delete incomplete multipart uploads after 3 days

For a versioning-enabled bucket, on the first day, the object count will increase by the number of current object versions, when the corresponding delete markers are created. All the previously current object versions become noncurrent. The next day, all the noncurrent object versions will be deleted permanently, but the delete markers will remain. On the third day, the delete markers will disappear, since there will be no previous object versions to cover with "tombstones".

With large numbers of objects, it's quite typical for some of these operations to fail, so a small fraction of objects may be missed in each of these three stages. So you may still see some current object versions after the first day, some noncurrent objects after the second day, and some delete markers the third day. Also note that while the processing of lifecycle rules starts at midnight UTC, it won't complete instantly. It typically takes until the end of the day (in UTC time) until the processing for the day completes.

As the bot mentioned, by suspending versioning for the bucket, you can often speed up the process somewhat, since no additional versions are created, but if there are already noncurrent versions of many objects, it may not make much of a difference. All you need to do is put the three lifecycle rules above in place and wait for a few days for the buckets to be cleared. For billing purposes, the deletions will be considered as having completed when they were supposed to be, regardless of some of the deletions getting delayed by a day or few.

The issue you're experiencing with deleting buckets containing millions of objects is complex, especially with versioning enabled. Here's an explanation of what's happening and some potential solutions:

1. Versioning Impact: When versioning is enabled, deleting an object doesn't actually remove it but creates a delete marker. This explains why your object count increased while the total size remained the same. Each delete operation essentially created a new version (delete marker) for each object.

2. Lifecycle Rules: Your current lifecycle rules are a good start, but they might not be sufficient for versioned buckets with many delete markers.

3. Suspending Versioning: Suspending versioning now won't affect existing versions but will prevent new versions from being created. It's a good step but won't solve the existing problem.

To resolve this issue and delete the buckets, consider the following approach:

1. Keep versioning suspended to prevent new versions from being created.

2. Modify your lifecycle rules:

   • Keep the rule to expire current versions of objects after 1 day.
   • Add a rule to permanently delete noncurrent versions of objects after 1 day.
   • Add a rule to delete expired object delete markers.

3. Wait for the lifecycle rules to take effect. This may take several days for buckets with millions of objects.

4. If the lifecycle rules don't completely empty the bucket after a few days, you may need to use the AWS CLI or SDK to programmatically delete remaining objects and versions.

5. For future reference, when deleting a bucket that contains objects and doesn't have versioning enabled, you can use the AWS CLI 'rb' command with the '--force' parameter to delete the bucket and all its contents in one operation.

If these steps don't resolve the issue, you may need to contact AWS Support for further assistance, as there could be underlying issues specific to your account or buckets.

Remember, deleting millions of objects can take a significant amount of time and may incur data transfer costs, so monitor the process closely.

Sources
Deleting a bucket - Amazon Simple Storage Service
Deleting objects from versioning-suspended buckets - Amazon Simple Storage Service
DeleteObject - Amazon Simple Storage Service