Does AWS Datasync fail if the file is modified at the same time it is being transferred from source to destination?

0

In some cases, during the hourly schedule for the DataSync task to transfer about 20 files results in an error status and I believe it is due to the files being updated at the same time of the transfer task. What would be the workaround for this?

Ra
asked 2 months ago136 views
2 Answers
1

Please refer to this documentation regarding Open and Locked files:

Open and locked files

  • Keep in mind the following when trying to transfer files that are open (in use) or locked:

  • In general, DataSync can transfer open files without any limitations.

  • If a file is open and being written to during a transfer, DataSync can detect this kind of inconsistency during the transfer task's verification phase. To get the latest version of the file, you must run the task again.

  • If a file is locked and the server prevents DataSync from opening it, DataSync skips the file during the transfer and logs an error.

  • DataSync can't lock or unlock files.

profile pictureAWS
EXPERT
answered 2 months ago
  • I see, this makes sense but any idea on a work around then since this will cause errors?

0

If files are modified during a DataSync task, the task will complete and end with failures indicating verification errors. DataSync task verification options validate the data on the destination matches the source [1]. You can rerun the DataSync task with the task option Transfer only data that has changed [2] to compare the source and destination data set and transfer files that do not match. The next hourly scheduled task will also prepare and transfer files that do not match.

[1] https://docs.aws.amazon.com/datasync/latest/userguide/configure-data-verification-options.html

[2] https://docs.aws.amazon.com/datasync/latest/userguide/configure-metadata.html

AWS
answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions