By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

S3 Bucket not deleting files after retention period passed.

0

Recently noticed our S3 bucket was extremely large compared to our Backup size. Items in the bucket are being "marked deleted" but not being removed once the life cycle rule age has passed.

Veeam Backup and Replication server is tagging old backups as deleted however they are not being removed as Object lock is enabled.

S3 Bucket Object lock is enabled. retention date is on Lifecylce rule is created to apply to all objects in bucket Lifecycle rule actions: Delete expired object delete markers or incomplete multipart uploads permanently delete items over 400 days old

Is there a way to force a deletion for mark deleted items to reduce space on our AWS Bucket while keeping our Current backup items?

Topics
Storage
Tags
Amazon Simple Storage ServiceS3 Object Lock
Language
English
Austin
asked 8 months ago897 views
2 Answers
1

It'd be really helpful, if you could add a snapshot of your lifecycle policy.

Delete expired object delete markers: You cannot enable Delete expired object delete markers if you enable Expire current versions of objects. This action is not available when "Expire current versions of objects" is selected.

Current version of object would be expired in 'n' days through rule Expire current versions of objects but you would not want to keep non-current version of objects there for longer duration, so based on your requirement I'd suggest you to add days in the rule Permanently delete noncurrent versions of objects accordingly, like how many days you want to keep noncurrent version objects to stay there. If you want to keep noncurrent version for 1 day then enter day value as 1 for this rule.

Additional Reference:

Hope you find this helpful.

profile pictureAWS
EXPERT
secondabhi_aws
answered 8 months ago
  • Austin
    8 months ago

    Thank you for your reply, Anything with the "Delete marker" is a non current version correct? So i could set this at 30 days after its marked and it would be removed?

    Here is a link to an image of how i have the life cycle configured currently. https://send.tresorit.com/a#fDLn2FKOwwa1g8qbf7kLTw

    Thank you

  • secondabhi_aws EXPERT
    8 months ago

    Noncurrent version Object: Whenever a new version of an existing object is uploaded in a version enabled bucket, all versions of this object except just uploaded one, would be noncurrent versions of object. Delete marker is a different topic, an object would have delete marker, if we delete the object in a version enabled bucket, that deleted object would still be there but with delete marker. Please refer delete marker documentation which I added in my answer.

    Here are some other useful blogs/re:Post article:

    How to setup Lifecycle rule for S3 noncurrent version of objects

    Reduce storage costs with fewer noncurrent versions using Amazon S3 Lifecycle

    I've included these artifacts in my answer as well.

    Feel free to comment here, happy to help further.

  • Austin
    8 months ago

    Ok thank you, Will lifecycle rules delete the objects within the bucket while Object lock is enabled after the retention period has passed? With the lifecycle rule set to 60 days, all non current objects have a delete marker will be permanently deleted from the bucket. is this correct?

  • secondabhi_aws EXPERT
    8 months ago

    Sorry Austin for delayed response. That's a great question, no lifecycle policy won't delete the objects with object lock enabled(compliance mode) however as that object lock expires and if that objects fall under lifecycle rule expiration, it'd then expire. But in Governance mode, this can be changed. No, non current object won't necessarily have delete marker. As I mentioned earlier, suppose I uploaded an object 'a.txt', then I updated this file at local and uploaded again to s3 same location, now if S3 bucket is version enabled then previously uploaded 'a.txt' would be non-current version of object 'a.txt' but it won't have delete marker. Delete marker gets attached to an object, when you delete the object. Now with the same example, after uploading updated 'a.txt', you have two versions, one is current and one noncurrent, if you now delete this 'a.txt', then a delete marker would be attached with a.txt.

    Hope this explains and provide you the clarity. Please feel free to comment here if you have further questions, happy to help.

  • secondabhi_aws EXPERT
    8 months ago

    Do you have any additional questions, happy to help?

1

It also depends on the mode that has been selected for Object Lock.
If you are using Compliance, the objects are locked until the expiration date and cannot be shortened or change the mode.
If the objects are locked via Governance mode, the changes (overwrite / lock settings) can be made by user with special permissions. Those users can perform a version specific delete request on the non-current versions and delete markers.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-overview.html#object-lock-retention-modes

AWS
rajesh pichaimani
answered 8 months ago
  • Austin
    8 months ago

    Hello, Our Object lock settings are currently set to the following

    "Object Lock Enabled

    Default retention Automatically protect new objects put into this bucket from being deleted or overwritten. Enabled

    Default retention mode Compliance

    Default retention period 100 days"

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content