By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

What is the command/how to sync directly to flexible glacier?

0

I use AWS as an off-site emergancy-in-case-the-house-burns-down back-up. I deliberately don't want it to automatically sync to the cloud (since if I frack something up, I like to have something from an older back-up just in case).

I perform this back up peroidically (a few times a year) back-up via synchronisation my data to AWS via the Powershell interface. However, as I have a lot of files and I move them around sometimes, It can get quite tedious to manually search through and change all the changed/new/updated files from Standard to Glacier Flexible Retrieval. This is especially time-consuming when dealing with the folders which cannot be sorted.

I have avoided generally doing whole folders, since I am under the impression that that does count towards the file usage and I don't want to be wasting money (I am on an extremely tight budget) changing something that's glacier to glacier.

I am currently using the command line (which was provided to me by someone, as my understanding of the command lines is negligable) as follows (for example):

aws s3 sync Documents s3://[bucket]/Documents

How (or even can I) change this command line so that it syncs directly the flexible glacier, and thus eliminate this part of the back-up? I presume, if possible, it would mean changing the "s3" parts, but I wouldn't know exactly what to.

(Or am I whittling about nothing and could, in fact, do it by folders?)

Any help would be greatly appreciated.

Topics
StorageManagement & Governance
Tags
Amazon S3 GlacierAWS Command Line Interface
Language
English
Aotrs Commander
asked a year ago447 views
1 Answer
0

The s3 sync command doesn't support Glacier (mainly because you can't copy files directly from Glacier back to your local PC, you need to restore to S3-Standard storage first so a 2-way sync doesn't work).

The easiest solution would be to use your current approach to sync but configure a Lifecycle Rule to automatically move files within the appropriate destination S3 directory to Glacier Flexible Retrieval. You can set the Days after object creation value in the rule to 0 to ensure this happens immediately and restrict this rule to only operate on certain folders/prefixes.

This essentially automates the step in your current process where you are manually changing the storage class of newly uploaded objects to be Glacier Flexible Retrieval.

You should be aware that there are cost considerations in transitioning data from S3-Standard to S3 Glacier Flexible Retrieval which are detailed here. It is particularly worth noting that moving small files (under 128KB) to Glacier is not generally cost-efficient due to the minimum billable object size of 128KB plus the additional 40KB of metadata overhead. If you want to store many small files, consider combining them into a single, larger archive first.

profile pictureAWS
CarlP
answered a year ago
  • Aotrs Commander
    a year ago

    Right, thank you. I wish I had known the 128 thing before doing my latest back-up, but its probably now not worth sorting out. (Though the vast majority of my files are above that threshold.) I have set the rule up to automatically transition files of 128kb and larger for future back-ips. When I next do the next one (in... May? after the 90-days passes) I will perhaps have a purge of the small files on AWS before I do the re-sync, such that they will be on standard.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content