How does DataSync determine if a file has changed?
Example use case is a daily data export locally that needs to sync to S3. There is no way to check what has changed before exporting, so must export the full dataset. Aim is to only upload what has changed to S3. DataSync seems like it will work (it will only be one-way transfer daily so choosing DataSync over Transfer or File Gateway) but how does it determine what has changed?
I found the docs which say:
In the PREPARING status, DataSync examines the source and destination file systems to determine which files to sync. It does so by recursively scanning the contents of the source and destination file systems for differences.
Will this scan the contents of the files themselves, or just the filesystem metadata? Anyone know anymore details on how it determines what has changed?
- 최신
- 최다 투표
- 가장 많은 댓글
This page says
When a task starts, DataSync performs a recursive directory listing to discover all files and file metadata in the source and destination file system. These listings are used to identify differences and determine what to copy, and usually takes between a few minutes to a few hours.
Which suggests it's based on properties such as modification time and size, rather than a checksum.
