By using AWS re:Post, you agree to the AWS re:Post Terms of Use

How do I use a lifecycle configuration rule to empty an Amazon S3 bucket?

3 minute read
3

I have an Amazon Simple Storage Service (Amazon S3) bucket that stores millions of objects. I want to use a lifestyle configuration rule to empty the bucket so that I'm not charged for storage.

Resolution

Important: The following resolution permanently deletes all data in your Amazon S3 bucket. Because you can't recover the data, review all objects and data in the bucket before you delete them.

To create a lifecycle configuration rule that expires current versions of objects and permanently deletes previous versions of objects, complete the following steps:

  1. Open the Amazon S3 console.
  2. From the list of buckets, choose the bucket that you want to empty.
  3. Choose the Management tab.
  4. Choose Create lifecycle rule.
  5. For Lifecycle rule name, enter a rule name.
  6. For Choose a rule scope, choose Apply to all objects in the bucket.
  7. Select I acknowledge that this rule will apply to all objects in the bucket.
  8. For Lifecycle rule actions, select the following options:
    Expire current versions of objects
    Permanently delete noncurrent versions of objects
    Delete expired object delete markers or incomplete multipart uploads
  9. In the Expire current versions of objects field, for the Days after object creation field, enter 1.
  10. In the Permanently delete noncurrent versions of objects field, for the Days after objects become noncurrent field, enter 1.
  11. To delete all versions, leave the Number of newer versions to retain (Optional) field empty.
  12. Select Delete incomplete multipart uploads, and then enter 1 for the Number of days field.
  13. Choose Create rule.
  14. To create a second lifecycle rule, repeat steps 4-7.
  15. Select the following option: Delete expired object delete markers or incomplete multipart uploads.
  16. Select Delete expired object delete markers.
  17. Choose Create rule.

Amazon S3 runs lifecycle rules once a day. After the first time that Amazon S3 runs the rules, all objects that are eligible for expiration are marked for deletion. You're no longer charged for objects that are marked for deletion.

Amazon S3 asynchronously expires object versions and removes delete markers. It might take the rules a few days to run before the bucket is empty. For more information about asynchronous object removal in Amazon S3, see Expiring objects.

You can also use the AWS Command Line Interface (AWS CLI) to create a lifecycle rule to empty your S3 bucket.

JSON Example:

{
    "Rules": [{
            "Expiration": {
                "Days": 1
            },
            "ID": "FullDelete",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "NoncurrentVersionExpiration": {
                "NoncurrentDays": 1
            },
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 1
            }
        },
        {
            "Expiration": {
                "ExpiredObjectDeleteMarker": true
            },
            "ID": "DeleteMarkers",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled"
        }
    ]
}

Related information

Removing expired object delete markers

Managing your storage lifecycle

How do I delete Amazon S3 objects and buckets?

Deleting a bucket

Setting lifecycle configuration on a bucket

AWS OFFICIAL
AWS OFFICIALUpdated a year ago
9 Comments

I want to empty a bucket using the AWS SDK for JavaScript. According to the documentation, "If you want the S3 Lifecycle rule to apply to all objects in the bucket, specify an empty prefix". However, I'm not sure how to do this. When I don't include a "Prefix" or a "Filter" property or include a "Filter" with an undefined "Prefix", I receive an error response of "MalformedXML: The XML you provided was not well-formed or did not validate against our published schema". The rule I'm trying to send is shown below:

const params = {
    Bucket: "bucket-name",
    LifecycleConfiguration: {
        Rules:  {
            ID: "Empty bucket s3",
            Filter: {
            }
            Status: "Enabled",
            Expiration: {
                Days: 1
            },
            AbortIncompleteMultipartUpload: {
                DaysAfterInitiation: 1
            }
        }
};
replied 2 years ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
EXPERT
replied 2 years ago

This is not possible using boto3, the example:

    lifecycle_configuration = {
        'Rules': [
            {
                'Expiration': {
                    'Days': 1,
                },
                'ID': 'DeleteAfter1Days',
                'Status': 'Enabled',
                'Prefix': '',
                'NoncurrentVersionExpiration': {
                    'NoncurrentDays': 1
                },
                'AbortIncompleteMultipartUpload': {
                    'DaysAfterInitiation': 1
                }
            },
            {
                'Expiration': {
                    'ExpiredObjectDeleteMarker': True
                },
                'ID': 'ExpireDeleteMarkers',
                'Status': 'Enabled',
                'Prefix': '*'
            }
        ]
    }
    print(f'Creating lifecyle for {bucket_name}')

    s3.put_bucket_lifecycle_configuration(
        Bucket=bucket_name,
        LifecycleConfiguration=lifecycle_configuration
    )

will result in "An error occurred (InvalidRequest) when calling the PutBucketLifecycleConfiguration operation: Found overlapping prefixes '*' and '' for same action type 'Expiration'"

replied 2 years ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 2 years ago

I did this for my bucket, but it's been 2 days now and there is still data in my bucket? Does the rule not run automatically everyday?

replied a year ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied a year ago

In the past, if you had a large number of objects in a bucket, you could get a very large bill for LIST API calls before the DELETE calls. Can you verify that if the Lifecycle Policy above is used that you will not get a large bill for deleting millions/billions of objects?

replied 9 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
EXPERT
replied 9 months ago

Did we find any solution to make it work in boto3?

replied 2 months ago