Sync aws s3 glacier files not checking timestamp of file size but only file existence on destination ?

0

How to sync aws s3 glacier files not checking timestamp of file size but only file existence on destination ?

We regularly backed up our picture source files (it's about a couple of hundred thousands) to AWS glacier, using --size-only and --delete option (as timestamp of the files could be differ depending of the fact that editing software change timestamp every time it has been accessing a file). So that files only edited, so differed in size were synced, as well as those files that were no more existing on source were deleted on s3 destination. But recently when we changed the editing software version it had to upgrade its catalog and, much worse, while updating it wrote to some files such as JPG or DNG some inner metadata, which itself led to change in their size. Consequently, now --size-only option is not enough, it now doesn't matter at all as AWS tries to sync about 80% of files that were not updated by editing and don't have any actual differences on source, but about of metadata adding has tiny differences in size. In fact it looks like we have to re-upload 80% of the files again, without any reasons to.

So I wonder, is there a possibility to sync local and s3 buckets based only on the existence of a file, not about its timestamp of size ?

grynn
質問済み 1年前181ビュー
1回答
0

Hi Grynn,

The s3 sync command updates any files that have a size or modified time that are different from files with the same name at the destination.

https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-objects-sync

You could explore using the --exclude and --include parameter flags with the s3 sync command

https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

profile pictureAWS
Abi-P
回答済み 8ヶ月前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ