Three buckets from three seperate sites (same account) all stopped syncing changes. CLI sync runs, sits, finishes, no output or changes.

0

I have three buckets under one account. For each one a server runs a sync via CLI V2 to it's dedicated bucket. I noticed that newer files weren't showing up. If I run aws s3 sync D:\MyFolder s3://mybucket, it will think about it for 5-10 minutes and finish with no output. I checked my credentials and region. No issues there. I can cp and rm files and folders in all three buckets. So access is there. It's like it runs and doesn't see changes. For the smallest bucket, I emptied, deleted, and recreated. That server was able to begin successfully syncing again. Bucket sizes are between 450GB-700GB. If I output the --debug information to a text file, I get an 85MB file with no clear errors. It's parsing the objects. I'm not sure what to look at next. I'd rather not delete and re-upload. Especially if this can happen again. Any guidance would be greatly appreciated. It's worth mentioning that that 85MB file with al the parsing was run against one folder with one item in it. Like it's parsing the whole bucket despite the narrow scope.

已提問 2 年前檢視次數 247 次
1 個回答
1

aws s3 sync by default will ignore files of same size and same timestamps if the object exists on both source and destination.

Sync will copy files from source to destination if:

  • if the size of the local file is different than the size of the s3 object
  • the last modified time of the local file is newer than the last modified time of the s3 object
  • the local file does not exist under the specified bucket and prefix.

Is it possible that the objects you are expecting to sync are same size/timestamp and therefore being skipped by s3 sync?

Flags of interest to help with modifying aws s3 sync behavior

--size-only (boolean) Makes the size of each key the only criteria used to decide whether to sync from source to destination.

--exact-timestamps (boolean) When syncing from S3 to local, same-sized items will be ignored only when the timestamps match exactly. The default behavior is to ignore same-sized items unless the local version is newer than the S3 version.

See also https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

AWS
已回答 2 年前
  • Thanks for the info. There have been a number of changed files and added folders. The changes are there, but not being recognized. I'll review the flags and see if any might help with troubleshooting.

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南