Three buckets from three seperate sites (same account) all stopped syncing changes. CLI sync runs, sits, finishes, no output or changes.

0

I have three buckets under one account. For each one a server runs a sync via CLI V2 to it's dedicated bucket. I noticed that newer files weren't showing up. If I run aws s3 sync D:\MyFolder s3://mybucket, it will think about it for 5-10 minutes and finish with no output. I checked my credentials and region. No issues there. I can cp and rm files and folders in all three buckets. So access is there. It's like it runs and doesn't see changes. For the smallest bucket, I emptied, deleted, and recreated. That server was able to begin successfully syncing again. Bucket sizes are between 450GB-700GB. If I output the --debug information to a text file, I get an 85MB file with no clear errors. It's parsing the objects. I'm not sure what to look at next. I'd rather not delete and re-upload. Especially if this can happen again. Any guidance would be greatly appreciated. It's worth mentioning that that 85MB file with al the parsing was run against one folder with one item in it. Like it's parsing the whole bucket despite the narrow scope.

질문됨 2년 전247회 조회
1개 답변
1

aws s3 sync by default will ignore files of same size and same timestamps if the object exists on both source and destination.

Sync will copy files from source to destination if:

  • if the size of the local file is different than the size of the s3 object
  • the last modified time of the local file is newer than the last modified time of the s3 object
  • the local file does not exist under the specified bucket and prefix.

Is it possible that the objects you are expecting to sync are same size/timestamp and therefore being skipped by s3 sync?

Flags of interest to help with modifying aws s3 sync behavior

--size-only (boolean) Makes the size of each key the only criteria used to decide whether to sync from source to destination.

--exact-timestamps (boolean) When syncing from S3 to local, same-sized items will be ignored only when the timestamps match exactly. The default behavior is to ignore same-sized items unless the local version is newer than the S3 version.

See also https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

AWS
답변함 2년 전
  • Thanks for the info. There have been a number of changed files and added folders. The changes are there, but not being recognized. I'll review the flags and see if any might help with troubleshooting.

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠