Three buckets from three seperate sites (same account) all stopped syncing changes. CLI sync runs, sits, finishes, no output or changes.

Question

I have three buckets under one account. For each one a server runs a sync via CLI V2 to it's dedicated bucket. I noticed that newer files weren't showing up. If I run aws s3 sync D:\MyFolder s3://mybucket, it will think about it for 5-10 minutes and finish with no output. I checked my credentials and region. No issues there. I can cp and rm files and folders in all three buckets. So access is there. It's like it runs and doesn't see changes. For the smallest bucket, I emptied, deleted, and recreated. That server was able to begin successfully syncing again. Bucket sizes are between 450GB-700GB. If I output the --debug information to a text file, I get an 85MB file with no clear errors. It's parsing the objects. I'm not sure what to look at next. I'd rather not delete and re-upload. Especially if this can happen again. Any guidance would be greatly appreciated. It's worth mentioning that that 85MB file with al the parsing was run against one folder with one item in it. Like it's parsing the whole bucket despite the narrow scope.

Answer

aws s3 sync by default will ignore files of same size and same timestamps if the object exists on both source and destination.

Sync will copy files from source to destination if:
+ if the size of the local file is different than the size of the s3 object
+ the last modified time of the local file is newer than the last modified time of the s3 object
+ the local file does not exist under the specified bucket and prefix.

Is it possible that the objects you are expecting to sync are same size/timestamp and therefore being skipped by s3 sync?

Flags of interest to help with modifying aws s3 sync behavior

```
--size-only (boolean) Makes the size of each key the only criteria used to decide whether to sync from source to destination.

--exact-timestamps (boolean) When syncing from S3 to local, same-sized items will be ignored only when the timestamps match exactly. The default behavior is to ignore same-sized items unless the local version is newer than the S3 version.
```

See also https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

Three buckets from three seperate sites (same account) all stopped syncing changes. CLI sync runs, sits, finishes, no output or changes.

Relevant content