By using AWS re:Post, you agree to the AWS re:Post Terms of Use

S3 Bucket Delete Marker Policy Not Working

0

Hi, I have a S3 bucket with a lifecycle policy and it was version enabled. The policy was to delete objects after 7 days and due to versioning the objects had a delete marker. Now I have created a new lifecycle policy around 3-4 days back to remove the delete markers permanently as I don't need the files anymore and when I checked the bucket today as well, the objects are still there. Please tell me how can I remove the objects as I have around 100 TB data and I cannot remove it manually. Please help me Thanks

asked 3 years ago2.2K views
4 Answers
0

Hi

If you would like to remove all objects inside bucket you can use EMPTY feature in S3 to clear all objects in S3:

You can empty a bucket's contents using the Amazon S3 console, AWS SDKs, or AWS Command Line Interface (AWS CLI). When you empty a bucket, you delete all the objects, but you keep the bucket. After you empty a bucket, it cannot be undone. When you empty a bucket that has S3 Bucket Versioning enabled or suspended, all versions of all the objects in the bucket are deleted.

aws s3 rm s3://bucket-name/doc --recursive

** Delete Markers:**

If you would like to use delete markers to delete all versions of the object you can use the below commnad

aws s3api delete-object --bucket DOC-EXAMPLE-BUCKET1 --key test.txt --version-id versionID

AWS Docs Information:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeletingObjectVersions.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/empty-bucket.html

If you have any questions please let me know

GK

profile picture
EXPERT
answered 3 years ago
0

Use lifecycle policies to remove older versions leaving only delete markers and use rules to have them removed too. Checkout Example 7. Hope that helps
https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html#lifecycle-config-conceptual-ex7

answered 3 years ago
  • Thanks Sandeep for your response. Just a quick question, if I set the policy to Permanently Delete Expired Marker Objects then it will not be touching any other object in the bucket except the ones with Delete Markers right?

0

The action of automatically removing the delete marker permanently through S3 Lifecycle Policy can take some time. So please wait for at least a week for the objects to disappear on your console.

Amazon S3 runs lifecycle rules once every day. After the first time that Amazon S3 runs the rules, all objects that are eligible for expiration are marked for deletion. You're no longer charged for objects that are marked for deletion. However, rules might take a few days to run before the bucket is empty because expiring object versions and cleaning up delete markers are asynchronous steps.

If the objects are still there after you wait a longer period of time, please update with me through this re:Post.

For additional information, please refer to the following re:Post pages: https://repost.aws/knowledge-center/s3-lifecycle-rule-delay https://repost.aws/knowledge-center/s3-empty-bucket-lifecycle-rule

answered a year ago
0

I've created a lifecycle rule:

Noncurrent versions actions
Day 0
Objects become noncurrent
Day 1
20 newest noncurrent versions are retained
All other noncurrent versions are permanently deleted

I created this rule on my bucket almost 2 weeks ago (that's roughly 14 day ago). The prefix contains a file with multiple versions accumulated every business day since roughly August 30th. When I goto the AWS Management Console, for S3 I notice I have roughly 95 pages of versions for the one file I'm trying to delete (and it goes all the way back to August 2023). I created an AWS support ticket 6 days (since I pay for support, but I'm here instead since support hasn't helped). Today I got a response from AWS Support:

Hello,

Thank you for reaching out to AWS Premium Support. This is Ezra, and I am pleased to assist you regarding this case.

=== Issue === From your message, I understand you are looking to know why the noncurrent versions didn't remove from your bucket after few days.

=== Reply === Depending on the previous correspondences, the operating of the lifecycle may be delay for several days. There is no guarantee of the time lifecycle will take to perform the operation. Please kindly allow the lifecycle to process asynchronously for a while. In this moment, we couldn't take any further action in addition to waiting.

If you would like to confirm whether the lifecycle is running, use Amazon SNS to subscribe lifecycle event (s3:LifecycleExpiration:Delete) and it will notify you when an object version is permanently deleted by an S3 Lifecycle configuration. Please refer [1] to know more.

I hope you find the above information is helpful. Please feel free to reach me in case you have further queries and I will be happy to help.

Have a warm day!

[1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html : Event notification types and destinations

We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.

Best regards, Ezra L. Amazon Web Services

I'm really confused by AWS support. Are they saying that my object versions should be deleted within a few days of when I created my bucket lifecycle rule? Or is AWS support saying it will take a few more days on top of the 14 days that my rule has existed? Or is AWS support saying there is no guaranty that lifecycle rule will perform the assigned operation? Or is AWS support saying that he/she/them cannot take any more action because they can't troubleshoot, and requires a few more days of just waiting to see what happens? The first 3/4 of the response seems very GPT-esque response. The last quarter of the support response seems a major BLAH. Really, I should create an event rule to detect when my rule is or not executing? Equivalently, I could say that they should either (1) reach out to internal team to determine why the rule hasn't completed since the 14 days it's been made, (2) for them to go look at CloudTrail (it shouldn't be my responsibility that AWS services is not working as intended).

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions