How does DataSync determine if a file has changed?

0

Example use case is a daily data export locally that needs to sync to S3. There is no way to check what has changed before exporting, so must export the full dataset. Aim is to only upload what has changed to S3. DataSync seems like it will work (it will only be one-way transfer daily so choosing DataSync over Transfer or File Gateway) but how does it determine what has changed?

I found the docs which say:

In the PREPARING status, DataSync examines the source and destination file systems to determine which files to sync. It does so by recursively scanning the contents of the source and destination file systems for differences.

Will this scan the contents of the files themselves, or just the filesystem metadata? Anyone know anymore details on how it determines what has changed?

AWS
asked 4 years ago918 views
1 Answer
0
Accepted Answer

This page says

When a task starts, DataSync performs a recursive directory listing to discover all files and file metadata in the source and destination file system. These listings are used to identify differences and determine what to copy, and usually takes between a few minutes to a few hours.

Which suggests it's based on properties such as modification time and size, rather than a checksum.

profile pictureAWS
EXPERT
James_S
answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions