Sync aws s3 glacier files not checking timestamp of file size but only file existence on destination ?

0

How to sync aws s3 glacier files not checking timestamp of file size but only file existence on destination ?

We regularly backed up our picture source files (it's about a couple of hundred thousands) to AWS glacier, using --size-only and --delete option (as timestamp of the files could be differ depending of the fact that editing software change timestamp every time it has been accessing a file). So that files only edited, so differed in size were synced, as well as those files that were no more existing on source were deleted on s3 destination. But recently when we changed the editing software version it had to upgrade its catalog and, much worse, while updating it wrote to some files such as JPG or DNG some inner metadata, which itself led to change in their size. Consequently, now --size-only option is not enough, it now doesn't matter at all as AWS tries to sync about 80% of files that were not updated by editing and don't have any actual differences on source, but about of metadata adding has tiny differences in size. In fact it looks like we have to re-upload 80% of the files again, without any reasons to.

So I wonder, is there a possibility to sync local and s3 buckets based only on the existence of a file, not about its timestamp of size ?

grynn
asked a year ago173 views
1 Answer
0

Hi Grynn,

The s3 sync command updates any files that have a size or modified time that are different from files with the same name at the destination.

https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-objects-sync

You could explore using the --exclude and --include parameter flags with the s3 sync command

https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

profile pictureAWS
Abi-P
answered 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions