S3 - change storage class of objects based on current storage class (only change STANDARD to GLACIER_IR)

Question

I had to migrate several hundred thousand objects (JPG files for photos) from one bucket to another. While using the cp command, I had assumed that since its set to copy the meta data, it would preserve the storage class. Wrong... a huge chunk of files defaulted to STANDARD storage.

Now I have about 1/3 of the objects in the new bucket stored STANDARD the other 2/3 in the ONEZONE_IA storage class... oops.

I set a lifecycle rule on the new bucket to transition all objects to GLACIER_IR after 1 day of upload. But it's been several days and the 1 terabyte sitting at STANDARD has not budged, according to the bucket metrics. I guess lifecycle rules apply to objects uploaded AFTER the rule is created.. or am I missing something? I set the lifecycle rule to apply with out a prefix limit, and to objects bigger than 128kb to test (I've heard that transitioning objects under 128k won't save much, since the overhead charge will keep the min $$ at 128kb rate).

If I run a cp command to replicate all objects to the new desired storage class, it will cost me hundreds of dollars in PUT API calls (learned that the hard way too). Wondering if there's a concise way to do this, or am I mis-understanding how lifecycle  rules.

Thank you in advance.

razguru · Answer

S3 Lifecycle rule runs once a day at midnight UTC and evaluates the rules on all objects in scope (not just the newly created objects post LC rule setup).
With just several hundred thousands objects and with days after creation = 1 rule, it shouldn't take more than 5 days ideally.  LC has an inbuilt mechanism to discard object <128KB in transition that means even if you hadn't set the min size limit condition these subset of objects would still be discarded and stayed there at your source class (Standard). 
You can leverage storage lens dashboard to see storage class distribution of your target bucket. Apply filter on bucket and look at under stage class tab. 
If LC rule is properly configured then you might have majority of objects <128KB that's why very few objects qualified for the transition and perhaps went unnoticed.

Leo K · Answer

Lifecycle rules start to get processed at midnight UTC every day, but the actual execution of the transitions can take several days. However, billing is based on which storage class the objects are supposed to be in, based on your configured lifecycle rules, even if implementing them takes time. Objects scheduled for transition from Standard to Glacier-IR at midnight on Thursday, for example, are charged at the Glacier-IR rate starting at midnight UTC on Thursday.

The bucket-level metrics are only updated once a day. You could just browse some of the objects in your bucket to see if they've been transitioned to Glacier-IR. Your object count is quite low, so despite the potential for delays, I'd generally expect that the majority of the objects, or even all of them, would've been transitioned by the end of the first day. If many or most objects have been transitioned, then I'd say it's safe to assume that your lifecycle rule is correctly configured and the metrics are just lagging behind. If none have been transitioned and are definitely in the Standard class, from which all transitions are valid, then there's probably something wrong with the lifecycle rule.

Note also that you can set a transition rule to be executed 0 days after the objects were created. Expiration rules have a minimum of 1 day.

S3 - change storage class of objects based on current storage class (only change STANDARD to GLACIER_IR)

Relevant content