How do I use a lifecycle configuration rule to empty an Amazon S3 bucket?

3 minute read
2

I have an Amazon Simple Storage Service (Amazon S3) bucket that stores millions of objects. I want to use a lifestyle configuration rule to empty the bucket so that I'm not charged for storage.

Resolution

Important: The following resolution permanently deletes all data in your Amazon S3 bucket. Because you can't recover the data, review all objects and data in the bucket before you delete them.

To create a lifecycle configuration rule that expires current versions of objects and permanently deletes previous versions of objects, complete the following steps:

  1. Open the Amazon S3 console.
  2. From the list of buckets, choose the bucket that you want to empty.
  3. Choose the Management tab.
  4. Choose Create lifecycle rule.
  5. For Lifecycle rule name, enter a rule name.
  6. For Choose a rule scope, choose Apply to all objects in the bucket.
  7. Select I acknowledge that this rule will apply to all objects in the bucket.
  8. For Lifecycle rule actions, select the following options:
    Expire current versions of objects
    Permanently delete noncurrent versions of objects
    Delete expired object delete markers or incomplete multipart uploads
  9. In the Expire current versions of objects field, for the Days after object creation field, enter 1.
  10. In the Permanently delete noncurrent versions of objects field, for the Days after objects become noncurrent field, enter 1.
  11. To delete all versions, leave the Number of newer versions to retain (Optional) field empty.
  12. Select Delete incomplete multipart uploads, and then enter 1 for the Number of days field.
  13. Choose Create rule.
  14. To create a second lifecycle rule, repeat steps 4-7.
  15. Select the following option: Delete expired object delete markers or incomplete multipart uploads.
  16. Select Delete expired object delete markers.
  17. Choose Create rule.

Amazon S3 runs lifecycle rules once a day. After the first time that Amazon S3 runs the rules, all objects that are eligible for expiration are marked for deletion. You're no longer charged for objects that are marked for deletion.

Amazon S3 asynchronously expires object versions and removes delete markers. It might take the rules a few days to run before the bucket is empty. For more information about asynchronous object removal in Amazon S3, see Expiring objects.

You can also use the AWS Command Line Interface (AWS CLI) to create a lifecycle rule to empty your S3 bucket.

JSON Example:

{
    "Rules": [{
            "Expiration": {
                "Days": 1
            },
            "ID": "FullDelete",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "NoncurrentVersionExpiration": {
                "NoncurrentDays": 1
            },
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 1
            }
        },
        {
            "Expiration": {
                "ExpiredObjectDeleteMarker": true
            },
            "ID": "DeleteMarkers",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled"
        }
    ]
}

Related information

Removing expired object delete markers

Managing your storage lifecycle

How do I delete Amazon S3 objects and buckets?

Deleting a bucket

Setting lifecycle configuration on a bucket

AWS OFFICIAL
AWS OFFICIALUpdated 4 months ago
6 Comments

I want to empty a bucket using the AWS SDK for JavaScript. According to the documentation, "If you want the S3 Lifecycle rule to apply to all objects in the bucket, specify an empty prefix". However, I'm not sure how to do this. When I don't include a "Prefix" or a "Filter" property or include a "Filter" with an undefined "Prefix", I receive an error response of "MalformedXML: The XML you provided was not well-formed or did not validate against our published schema". The rule I'm trying to send is shown below:

const params = {
    Bucket: "bucket-name",
    LifecycleConfiguration: {
        Rules:  {
            ID: "Empty bucket s3",
            Filter: {
            }
            Status: "Enabled",
            Expiration: {
                Days: 1
            },
            AbortIncompleteMultipartUpload: {
                DaysAfterInitiation: 1
            }
        }
};
replied 10 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 10 months ago

This is not possible using boto3, the example:

    lifecycle_configuration = {
        'Rules': [
            {
                'Expiration': {
                    'Days': 1,
                },
                'ID': 'DeleteAfter1Days',
                'Status': 'Enabled',
                'Prefix': '',
                'NoncurrentVersionExpiration': {
                    'NoncurrentDays': 1
                },
                'AbortIncompleteMultipartUpload': {
                    'DaysAfterInitiation': 1
                }
            },
            {
                'Expiration': {
                    'ExpiredObjectDeleteMarker': True
                },
                'ID': 'ExpireDeleteMarkers',
                'Status': 'Enabled',
                'Prefix': '*'
            }
        ]
    }
    print(f'Creating lifecyle for {bucket_name}')

    s3.put_bucket_lifecycle_configuration(
        Bucket=bucket_name,
        LifecycleConfiguration=lifecycle_configuration
    )

will result in "An error occurred (InvalidRequest) when calling the PutBucketLifecycleConfiguration operation: Found overlapping prefixes '*' and '' for same action type 'Expiration'"

replied 9 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 9 months ago

I did this for my bucket, but it's been 2 days now and there is still data in my bucket? Does the rule not run automatically everyday?

replied 3 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 3 months ago