Skip to content

Understanding S3 / Glacier Deep and is it safe to delete

0

Good afternoon. I am taking over an account and looking at their current storage. There are a lot of files in Glacier Deep Archive, when I drill down, I don't see any expire date, so a lot are simply not needed, but a while ago they had someone delete a bunch of files that cost thousands of dollars which naturally we want to avoid. There at 50+ buckets, most lifecycle rules have been removed for this cleanup, so basically I want to say anything > x purge. But before that naturally I want to make sure there not in the 180 day window.

So, if there is no expire date, but the files are labled 2022-X, I know they have been there a while, so would we incur charges to just purge?

I am trying to use metrics and storage lens to also help navigate and see all the real standard IA files, and address those, etc. but there must be a better console way or lifecycle to say if its > 180 purge, not get charged, etc. From a few posts, it appears that once a file has been in deep archive for 180 days I can delete with no charge, so I think I can set a lifecycle to say if storage class was glacier and > 180 days, delete. Am I correct?

Thanks all

asked a year ago2.2K views

2 Answers
0

You're on the right track with your understanding of S3 Glacier Deep Archive deletion policies. Here's what you need to know:

  1. Minimum storage duration: Objects in Glacier Deep Archive have a minimum storage duration of 180 days. If you delete objects before this period, you'll incur early deletion fees.

  2. Safe deletion: For objects that have been in Glacier Deep Archive for more than 180 days, you can delete them without incurring any additional charges beyond the normal S3 request fees.

  3. Identifying safe-to-delete objects: If the objects are labeled with dates from 2022 or earlier, they have likely been in storage for more than 180 days, making them safe to delete without extra charges.

  4. Using lifecycle rules: You are correct that you can set up a lifecycle rule to automatically delete objects that have been in Glacier Deep Archive for more than 180 days. This is an efficient way to manage your storage and avoid unnecessary costs.

  5. Caution with bulk deletions: When dealing with a large number of objects (millions or billions), it's recommended to use lifecycle rules for deletion rather than manual processes. This helps avoid unexpected costs and ensures a more controlled deletion process.

  6. Checking last access date: For objects in standard storage classes, you might want to consider their last access date before deletion. This can be done by enabling and analyzing S3 server access logs or CloudTrail data event logs.

  7. Gradual approach: Given the previous incident of costly deletions, it might be wise to start with a small subset of objects or a single bucket to verify your deletion strategy before applying it widely.

Remember, while DELETE operations themselves are free, you need to be cautious about the storage duration of the objects you're deleting. Always double-check your lifecycle rules and deletion criteria before implementing them across multiple buckets.
Sources
Delete Billions of objects from S3 Bucket | AWS re:Post
Permanently delete from Glacier Deep Archive | AWS re:Post
Delete unused files from S3 | AWS re:Post
S3 Early deletion Fees | AWS re:Post

answered a year ago

EXPERT

reviewed a year ago

0

Hello.

If you configure the following settings, you will have a rule that deletes objects 180 days after they are moved to Glacier.
a

As you are aware, deleting an object from Glacier before 180 days have passed since it was moved there incurs a charge.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-transition-general-considerations.html

Number of days you plan to keep objects archived – S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive are long-term archival solutions. The minimal storage duration period is 90 days for the S3 Glacier Flexible Retrieval storage class and 180 days for S3 Glacier Deep Archive. Deleting data that is archived to Amazon S3 Glacier doesn't incur charges if the objects you delete are archived for more than the minimal storage duration period. If you delete or overwrite an archived object within the minimal duration period, Amazon S3 charges a prorated early deletion fee. For information about the early deletion fee, see the "How am I charged for deleting objects from Amazon S3 Glacier that are less than 90 days old?" question on the Amazon S3 FAQ.

EXPERT

answered a year ago

EXPERT

reviewed a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.