By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Confused about permanently deleting objects in versioned s3 bucket

0

We have a versioned s3 bucket. I'd like to set it up so that we keep some number of non-current versions. I understand how to create that rule, but i don't understand how to ensure that the objects get permanently deleted once we have marked a versioned object as deleted.

to clarify what i am aiming for:

  • non deleted, current objects are never expired
  • if an object has not been marked as deleted, keep up to 10 non current versions and delete the remaining after 30 days
  • when an object has been marked as deleted, delete everything for the object after 30 days

the documentation has this example, which seems similar to what i am looking for:

<LifecycleConfiguration>
    <Rule>
       <ID>Rule 1</ID>
        <Filter>
          <Prefix>logs/</Prefix>
        </Filter>
        <Status>Enabled</Status>
        <Expiration>
           <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
        </Expiration>
        <NoncurrentVersionExpiration>     
            <NewerNoncurrentVersions>10</NewerNoncurrentVersions>
            <NoncurrentDays>30</NoncurrentDays>    
        </NoncurrentVersionExpiration>
    </Rule>
</LifecycleConfiguration>

however, my interpretation of that rule is that even once we mark an object as deleted, there would still be 10 noncurrent versions kept, and therefore the deletion markers would never expire. am i misunderstanding? is there some other way to keep a limited number of non current versions for non-deleted objects but expire (i.e. permanently delete) deleted objects after some time?

Topics
Storage
Tags
Amazon Simple Storage ServiceStorage
Language
English
james
asked 7 months ago1.2K views
2 Answers
1

A delete marker can be current or noncurrent. It becomes current when the object is removed. In this case you will have a delete marker that is current and some noncurrent versions. If you have a lifecycle rule that works on current objects, it will eventually mark the current delete marker as noncurrent which can later be expired by another lifecycle rule that works on noncurrent objects. There will be a situation where all object versions are deleted and only a single delete marker remains. In this case, it is considered as an expired delete marker. The rule that you have provided will remove that expired delete marker.

profile pictureAWS
EXPERT
AWS-User-alantam
answered 7 months ago
profile pictureAWS
EXPERT
iBehr
reviewed 7 months ago
  • james
    7 months ago

    i don't want to expire current non-deleted items. your answer seems to require there to be an expiration rule for any current item.

    is it possible you could outline the rules you have described above? maybe i'm just not following what you are trying to explain.

    i added some additional context to my question to clarify what i am trying to accomplish.

  • AWS-User-alantam EXPERT
    7 months ago

    Let's say you have an object created in a versioning enabled bucket.

    1. If there is an LC rule that deletes any current object after 30 days: 30 days pass by, the object is deleted and you'll see a delete marker that becomes current as well as a noncurrent version. But since you have this LC rule, the delete market will eventually become noncurrent after another 30 days. If you also have another LC rule that delete any noncurrent version after 30 days, the delete market will be eventually removed.
    2. If you don't have any LC rule that deletes any current object. You just manually delete the object. You'll still see a delete marker that becomes current as well as a noncurrent version. The only difference now is that the delete market will remain as current forever. Let's say you have a LC rule that removes any noncurrent version after 30 days. Another 30 days pass by, the noncurrent version will be permanently deleted. This will make the delete marker "expired" (even it's still "current") since there is nothing but just a delete marker. At this point, it becomes an expired delete marker. If the LC rule is also explicitly set to remove the expired delete marker, Amazon S3 will remove the delete marker.
  • james
    7 months ago

    so, i understand your explanation, and it matches my expectations. however, neither of those scenarios describe what i am trying to do. i detailed out what i am trying to achieve in my original question. i want to maintain some number of noncurrent items until the object is deleted. at that point, i want to be able to eventually expire/delete everything.

0

You'll need another Lifecycle Rule to delete the noncurrent versions of objects. And you can also specify the number of the newer versions to retain. Below are some blog posts than explains it in more detail.

https://repost.aws/knowledge-center/s3-lifecycle-rule-non-current-version

https://aws.amazon.com/blogs/storage/reduce-storage-costs-with-fewer-noncurrent-versions-using-amazon-s3-lifecycle/

profile pictureAWS
EXPERT
AWS-User-alantam
answered 7 months ago
profile pictureAWS
EXPERT
iBehr
reviewed 7 months ago
  • james
    7 months ago

    yes, i have a non-current expiration rule in what i posted above. that doesn't explain how to get the entire object eventually deleted once it is marked as deleted.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content