How does DataSync determine if a file has changed?

Question

Example use case is a daily data export locally that needs to sync to S3. There is no way to check what has changed before exporting, so must export the full dataset. Aim is to only upload what has changed to S3. DataSync seems like it will work (it will only be one-way transfer daily so choosing DataSync over Transfer or File Gateway) but how does it determine what has changed?

I found the [docs][1] which say:
> In the PREPARING status, DataSync examines the source and destination file systems to determine which files to sync. It does so by recursively scanning the contents of the source and destination file systems for differences.

Will this scan the contents of the files themselves, or just the filesystem metadata? Anyone know anymore details on how it determines what has changed?

[1]: https://docs.aws.amazon.com/datasync/latest/userguide/how-datasync-works.html#transfering-files

James S · Accepted Answer

[This page][1] says

> When a task starts, DataSync performs a recursive directory listing to discover all files and file metadata in the source and destination file system. These listings are used to identify differences and determine what to copy, and usually takes between a few minutes to a few hours.

Which suggests it's based on properties such as modification time and size, rather than a checksum.

[1]: https://docs.aws.amazon.com/datasync/latest/userguide/run-task.html

How does DataSync determine if a file has changed?

Adicione sua resposta

Conteúdo relevante