By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Unable to Delete objects from versioned bucket using Lifecycle Rule

0

Hi, I have been trying to setup a lifecycle rule to delete all the objects in my versioned bucket. My motive is to delete the complete bucket. Details

  1. The Bucket is versioned Bucket Properties

  2. Bucket is using around 120TB of storage all objects are older than 2 years the glacier transition was 1 year ago Bucket Metrics

  3. Lifecycle rule implemented (Currently i have implemented the following lifecycle rule but there is no change in the bucket size) Lifecycle rule implemented

The above lifecycle was implemented 4 weeks ago. I cant even see the objects in the bucket it takes forever to load Unable to see objects

Can someone please help me identify what am i doing wrong ? NOTE I want to delete the complete bucket but untill the bucket is empty AWS doesnt allow me to delete the bucket

Topics
StorageAWS Well-Architected Framework
Tags
Amazon Simple Storage ServiceAmazon S3 GlacierStorageCost Optimization
Language
English
Shithanshu Mishra
asked a year ago716 views
2 Answers
2
Accepted Answer

You have versioning enabled, but the lifecycle policy is only deleting the latest version objects. You should consider modifying your lifecycle policy to also delete the non-current (older) versions of the objects so that those will be deleted as well.

profile pictureAWS
EXPERT
Shlomo Dubrowin
answered a year ago
profile picture
EXPERT
Giovanni Lauria
reviewed 20 days ago
  • Shithanshu Mishra
    a year ago

    Hi Sholomo Dubrowin, Thank you for the response So i have to setup a Lifecycle policy with

    1. Expire current versions of objects
    2. Permanently delete noncurrent versions of objects
    3. Delete expired object delete markers or incomplete multipart uploads

    Noncurrent versions actions Day 0

    Objects become noncurrent

    Day 1

    0 newest noncurrent versions are retained
All other noncurrent versions are permanently deleted

    Delete expired object delete markers or incomplete multipart uploads Expired object delete markers

    Incomplete multipart uploads Delete after 1 days

    Sorry i am unable to share image in comments Does the above lifecycle looks good or should i make some more changes ?

  • Shlomo Dubrowin EXPERT
    a year ago

    Yes, that looks like it should do the trick. It may take a couple of days for the lifecycle to actually do the delete. It depends on what time the lifecycle runs and when the new policy is installed. But you should see a decrease in the number of objects in a couple of days.

    I see you shared CloudWatch metrics with the object counts, are you aware of the S3 Storage Lens, which is free with the default settings? Storage Lens can also show Multipart Upload metrics and noncurrent version metrics along with others.

    https://aws.amazon.com/blogs/aws/s3-storage-lens/

  • Shithanshu Mishra
    a year ago

    Hi Sholomo Dubrowin, Thank you for the answer i was able to delete the objects using your feedback.

0

I've created a lifecycle rule:

Noncurrent versions actions
Day 0
Objects become noncurrent
Day 1
20 newest noncurrent versions are retained
All other noncurrent versions are permanently deleted

I created this rule on my bucket almost 2 weeks ago (that's roughly 14 day ago). The prefix contains a file with multiple versions accumulated every business day since roughly August 30th. When I goto the AWS Management Console, for S3 I notice I have roughly 95 pages of versions for the one file I'm trying to delete (and it goes all the way back to August 2023). I created an AWS support ticket 6 days (since I pay for support, but I'm here instead since support hasn't helped). Today I got a response from AWS Support:

Hello,

Thank you for reaching out to AWS Premium Support. This is Ezra, and I am pleased to assist you regarding this case.

=== Issue === From your message, I understand you are looking to know why the noncurrent versions didn't remove from your bucket after few days.

=== Reply === Depending on the previous correspondences, the operating of the lifecycle may be delay for several days. There is no guarantee of the time lifecycle will take to perform the operation. Please kindly allow the lifecycle to process asynchronously for a while. In this moment, we couldn't take any further action in addition to waiting.

If you would like to confirm whether the lifecycle is running, use Amazon SNS to subscribe lifecycle event (s3:LifecycleExpiration:Delete) and it will notify you when an object version is permanently deleted by an S3 Lifecycle configuration. Please refer [1] to know more.

I hope you find the above information is helpful. Please feel free to reach me in case you have further queries and I will be happy to help.

Have a warm day!

[1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html : Event notification types and destinations

We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.

Best regards, Ezra L. Amazon Web Services

I'm really confused by AWS support. Are they saying that my object versions should be deleted within a few days of when I created my bucket lifecycle rule? Or is AWS support saying it will take a few more days on top of the 14 days that my rule has existed? Or is AWS support saying there is no guaranty that lifecycle rule will perform the assigned operation? Or is AWS support saying that he/she/them cannot take any more action because they can't troubleshoot, and requires a few more days of just waiting to see what happens? The first 3/4 of the response seems very GPT-esque response. The last quarter of the support response seems a major BLAH. Really, I should create an event rule to detect when my rule is or not executing? Equivalently, I could say that they should either (1) reach out to internal team to determine why the rule hasn't completed since the 14 days it's been made, (2) for them to go look at CloudTrail (it shouldn't be my responsibility that AWS services is not working as intended).

jeff1evesque
answered 7 months ago
  • jeff1evesque
    7 months ago

    My ticket with AWS support is about 2 weeks old, and my lifecycle rule is approximately 3 weeks old now. The one file I want cleaned up now has about 120 pages of versions. One of the main reasons I created the ticket with support was due to optimization. The more versions you have of a file, the slower it becomes reading the latest version. Support engineer told me that if versions are deleted, I won't get charged (though that was never any mention in the ticket). I told support that I don't care about being charged or what not, I just want things deleted (due to optimization). But, a few days ago support was nice enough to tell me they were going to reach out to internal team, and that I won't get charged for anything that has been deleted. Guess that little bit of trivia fact (you don't get charged for something you don't have) is about as useful as having 120 pages of versions clearly visible in the AWS management console.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content